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ABSTRACT 

A computer-assisted instructional system to teach 
college students the computer language, AID (Algebraic Interpretive 
Dialogue) , two control programs, and data collected by the two 
control programs are described. It was found that although first 
response errors were often those of AID syntax, such errors were 
easily corrected. Secondly, while students easily learned the 
mechanics of the instructional system, they rarely used features that 
allowed student control. Thirdly, algebraic formulations of problems 
appeared to be more difficult than transforming algebraic expressions 
into AID commands. Finally, the students had the greatest difficulty 
understanding hierarchy of arithmetic operations, use of functions, 
and the execution sequence of AID commands. (Author/RH) 
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STUDENT PERFORMANCE IN COMPUTER- ASSISTED INSTRUCTION IN PROGRAMMING 



J. E. Friend, J. D. Fletcher, and R. C. Atkinson 
Stanford University 
Stanford, California 94305 



1. Introduction 

In 1967 the Institute for Mathematical Studies in the Social Sciences 
received a three-year grant* from the National Aeronautics and Space Admin- 
istration to do exploratory research in the optimization of instruction 
and to develop a practical course of study using computer-assisted in- 
struction (CAI). The course that was developed was a one-quarter college 
course in computer science (Friend & Atkinson, 1971)* Developmental 
testing of the course was accomplished using NASA personnel and Stanford 
students. The course was not the only product of this grant; an entire 
computer-assisted instructional system (Friend, 1971) was developed, with 
this course “ 3 the first application. 

The following year, the National Science Foundation provided funds** 
for a second, and very similar, application of the instructional system 
developed under the NASA funding. The second course was an introduction 
to computer programming for culturally-deprived high school students. 

This course taught the programming language, BASIC, whereas the first 
course taught AID. Both AID and BASIC are higher-order algebraic lan- 
guages analogous to ALGOL and FORTRAN. The main difference between the 



* NASA Grant NGR-05-020-244 
**NSF Grant No. GJ-443X 




1 



two courses is that the AID course was written for college students with 
a good background in algebra, and the BASIC course was written for high 
school students with low reading ability and little or no background in 
algebra. The BASIC course was adequately tested with a group of over 
100 students in an inner-city high school in San Francisco. 

In 1970* the Office of Naval Research awarded a research contract 
to the Institute to continue its program of basic reseach in instructional 
strategies using the already developed instructional system and courses 
as research tools. The main subject of study was to be the AID course, 
but data collected in connection with the use of the BASIC course was 
also to be used if appropriate. 

This report is a preliminary discussion of the research conducted 
under the ONR contract and is concerned only with the AID course. 

2. The Instructional System 

The course "Computer-assisted Instruction in Programming: AID" is 
an introductory course in computer science for community or junior 
college students with some background in high-school algebra. The course 
is completely self-contained and requires no supervision from a qualified 
instructor of programming. A brief student manual is supplied to supple- 
ment the instruction given by computer. 

The computer used is a Digital Equipment Corporation PDP-10 located 
at Stanford University and owned and operated by the Institute for Math- 
matical Studies in the Social Sciences. Connected to this computer by 
telephone lines are "Model-33" teletypewriters located in the schools 
and used for communicating with students. Instructions are printed on 
the teletypewriter terminal, and the student responds by typing his 



replies on the same terminal. Teletypewriter operation is simple and can 
be learned from short instructions printed in the student manual. 

Once the student has the teletypewriter in operation, all further 
instruction is given by computer under the control of a program known as 
INST. This program, which is the major component of the INSTRUCT system, 
interprets coded lessons providing individualized, tutorial instruction 
to the student. This instructional system and the method of programming 
lessons for it are described adequately by Friend (1971); and a detailed 
description will not be repeated here. 

The AID course uses most of the features of the INSTRUCT system. 

The course contains 50 lessons organized into seven "lesson blocks." 

Each lesson block contains five tutorial lessons, followed by a self- test 
and a general review. The 50th lesson is a concluding lesson independent 
of the lesson blocks. The structure of the main strand is shown in 
Figure 1. The lessons vary in length from 10 to 60 exercises depending 
upon the content. Lessons of average length require about one hour to 
complete. Lesson length is completely under student control, and a 
student may take a few exercises or several lessons in one sitting. 

One of the primary teaching strategies used in the course is the 
provision for student control of the sequence of instruction. Students 
may skip from any exercise in the course to any other exercise at any 
time, retracing their steps if they wish, or skipping lessons entirely. 
This strategy is intended to encourage the student to take responsibility 
for learning the concepts, not simply for progressing through a given 
set of exercises. Most college students are capable, and desirous, of 
assuming this responsibility, and the provision for student control of 
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instruction is assumed to provide motivation. Whether or not all students 
are motivated by this treatment and whether or not it is an effective 
strategy in terms of the amount of learning taking place remains to be 
tested. 

Because of this allowance for student control, the 50 lessons may be 
taken in any sequence. If the student does not exercise his prerogative 
for choosing the sequence, the lessons are automatically sequenced for 
him; and it is assumed that most students will, in fact, do the lessons 
in the order indicated. 

Besides the main strand of lessons, the course also contains review 
lessons, one for each of the tutorial lessons in the seven lesson blocks. 
These review lessons are also tutorial and cover the same concepts as do 
the lessons they are associated with. However, they present each concept 
from a slightly different viewpoint providing additional practice in the 
skills to be learned. In general, each lesson covers five or six related 
concepts. In review lessons, the student may review whichever concepts 
he wishes, in any order he chooses. In fact, he must, choose the order; 
there is no automatic sequencing provided by the program. At the end 
of each tutorial lesson, the student is asked if he wants to review any 
of the ideas covered in the lesson he has just completed. The student 
need not wait for these reminders, of course, since he can call for any 
review, or any exercise in any review, whenever he wishes. 

Also associated with each tutorial lesson is a summary of the .'Lesson, 
and the student is reminded at the end of each .'Lesson that summaries are 
available. Each summary is printed in an 8-1/2" X 1.1" form that can be 
torn off and saved by the student as a permanent record. 




In addition to the main strand of lessons , the reviews, and the 
summaries, there is a strand of "extra-credit" problems containing more 
difficult programming problems to be solved by the more capable students. 
It is recommended to instructors that the solutions of these problems be 
submitted for extra-credit if the course is graded. Not every lesson has 
associated extra-credit problems because they are not always appropriate 
to the subject matter. If there are such problems, the student is asked 
if he wants to try them at the end of the lesson. 

The relationship and sequence of lessons, summaries, reviews and 
extra-credit problems are illustrated in Figure 2. 

As described above, the main strand of lessons is divided into seven- 
lesson blocks, five tutorial lessons followed by a self- test and a general 
review. Lessons numbered 6, 13, 20, 27, 34, 4l, and 48 are self-tests. 
Lessons numbered 7 > 14, 21, 28, 35, 42, and 49 are general reviews. The 
self-tests are optional and students are told at the beginning of each 
test that the test is for their information only and may be skipped. 

The tests cover the concepts taught in the preceding five lessons and 
provide a good indication of weaknesses and areas requiring review. The 
general review which follows the self-test is also optional and is recom- 
mended for students who do poorly in the self-test. The general review 
is programmed to call the reviews for individual lessons as subroutines. 
For example, the student may review parts of Lessons 1, 3, and 4 and skip 
the review of Lessons 2 and 5 . If he wishes to review Lesson 2, he will 
take the same review he would have taken if he had chosen to review the 
lesson immediately after taking it, and he can decide which concepts from 
that lesson to review and in which order. 
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This review system is a gross method for providing individualized 
remediation. A more sensitive means of individualizing remediation is 
used within the lessons themselves where remedial sequences of exercises 
are given immediately to students who demonstrate an inadequate under- 
standing of the material being taught. The remedial sequences that are 
•imbedded within the lessons are not optional and are automatically pro- 
vided for students whose responses indicate a lack of understanding. 
Because of this automatic remediation, different students may receive 
different numbers of exercises in a given lesson. 

A student who makes an incorrect response to an exercise may not 
need an entire sequence of remedial exercises. He may profit from a 
single specific corrective message, pointing out the error and allowing 
him another try at the same problem. This kind of specific correction 
is used for most exercises in the course. Messages are provided, not 

for all possible incorrect responses, but for those incorrect responses 

judged to be most likely to occur. 

In line with the provision for student control of the sequence of 
instruction, there is also provision for student control of the amount 
of instruction. This is done by providing additional instruction for 
almost every exercise in the form of "hints" and sample correct answers. 
After the problem statement is typed, the student is free to request a 
hint by typing a question mark. For many problems, a sequence of three 

or four hints is provided. The student may request one hint, make an 

attempt to answer the exercise, then ask for a second hint, and so on. 
Also, the student may ask for the correct answer at any time, either 
before or after he makes a try at the problem. He does this by typing 
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two keys Simultaneously, the CTRL (control) key and the letter T. The 
answer will be printed, and he will proceed to the next exercise. 

In the tutorial lessons, there is no limit on the number of attempts 
that a student may make for an exercise. As soon as he makes a correct 
response he will be given the next exercise in sequence, but if he fails 
to respond correctly, he is given another chance. If he cannot do the 
problem at all, he can judge for himself whether to continue trying or 
whether to go on with the lesson; whenever he decides to proceed without 
giving a correct response he can do so by typing CTRL-T, thereby getting 
the correct answer and the next exercise. 

The CTRL key is also used in several other student, controls. CTRL-G 
is used when the student wants to jump to a different exercise; after 
he types CTRL-G, the computer will ask "Where to?" and he can type the 
lesson and problem number that he desires. CTRL-U is used to erase entire 
lines, and CTRL-Z is used to stop the instructional program and end the 
session. 

3 . The AID Interpreter 

The AID language was chosen as the subject of the course, not only 
because it is a useful algebraic programming language, but also because 
it is easily learned by a beginner. AID was developed, under the name 
of JOSS (Mark & Amerding, 1967 ; Shaw, 1965), by the Rand Corporation for 
use by scientists and engineers at Rand who needed an easily learned 
programming language that was capable of performing complex algebraic 
tasks. JOSS was later implemented for several other computers under a 
variety of names. 
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One of the advantages of AID as a beginner's language is that it is 
an interpreted, rather than compiled, language, which will act immedi- 
ately on direct commands given by the user. Because of this feature the 
syntax and use of many AID commands can be taught to the student before 
he is taught about stored programs. In the AID course, the use of AID 
is introduced in Lesson 2 and the concept of stored programs is not in- 
troduced until Lesson 10. 

A second advantage of AID, as compared to, say, FORTRAN or LISP, is 
that the syntax is a subset of English; commands are easily read as 
English imperative sentences. As an example, here is a simple AID program: 

1.1 SET X = 1.3074. 

1.2 SET Y = X/37.5. 

1.3 TYPE Y IF Y < .029. 

1.4 TYPE Y - .01 IF Y > .029. 

1.5 TYPE "EUREKA" IF Y = .029. 

As with most programming languages, AID is easier to read than to 
write, and considerable practice is needed by most students before they 
can produce a complete simple program like the above without error. 

However, they can begin to produce simple direct commands like 

SET Y = X/37 • 5 
or 

TYPE Y - .01 

on their first day. 

Though simple to learn, AID is nevertheless a powerful tool for 
algebraic tasks. Definitions of conditional functions and recursive 
functions are relatively simple, and there are a variety of useful 
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standard functions, such as the basic trigonometric, exponential, and 
logarithmic functions. Lists and matrices can be easily defined, although 
AID does not provide the standard matrix manipulation functions found in 
some other programming languages. 

Basic programming features, common to all programming languages, are 
available in AID. Variables and functions can be labeled, as can stored 
commands. There are branching commands, subroutine calls, and conditional 
clauses, and there are input and output commands for both disk and teletype, 

AID commands and programs are interpreted by a program called, aptly, 
the AID interpreter. The interpreter used by the students of this course 
is a program written by Digital Equipment Corporation, the manufacturer 
of the Institute's PDP-10 computer. 

The use of the AID interpreter is taught in the course, and the 
students are expected to use it frequently to solve problems given them 
in the lessons. Thus, the students taking this course will use two pro- 
grams: INST, the instructional program which talks about the AID language.,, 
and AID, the interpreter which uses the AID language. In a previous 
version of the course, the two programs were completely independent and 
the students were required to learn to start and stop both programs so 
that they could switch back and forth to do the programming problems. 

This method was feasible but awkward and time consuming. For the present 
version of the course, both programs were modified to permit easy access 
from the instructional program to the AID interpreter. Students can call 
the AID interpreter at any time simply by typing the word "AID," and can 
return to the instructional program by typing "INST. " This interface 
provides an additional advantage for research in that it supplies an 



easy way to cross-reference data collected by the two programs. It is 
now possible to pass information, invisibly to the student, from one pro- 
gram to the other; every time the student calls the AID interpreter, the 
student’s identity and current position in the course are passed to the 
AID interpreter so that data collected by AID can be keyed to data col- 
lected by INST. 



4 . Data 

Both the instructional program and the AID interpreter contain data- 
collection subroutines that enable them to store information about student 
responses as the student is working. These data are stored temporarily 
on the disk and later transferred to permanent tape storage. 

The instructional program records the following information with each 
student response.: 

1. Student number. (Each student is assigned a unique number when 
he first enrolls for any computer-assisted instruction offered 
by the Institute.) 

2 . Date . 

3. Time of day. 

4 . Lesson identifier. 

5* Problem number. 

6. Subproblem number • 

7* Trial number. 

8. Student response. (This is an exact character- by- character 
record of the response made by the student, excluding erasures 
made by the student.) 




12 



9* Analysis of correctness. (This is a record of ^ how the instruc- 
tional program scored the student's response to this exercise.) 

10. Lesson score. (This is a cumulative score of the student's 
first responses to exercises within this lesson.) 

11. Number of hints. (This is a record of the number of hints 
requested by the student before he made this reponse.) 

12. Answer provided by program? (This switch records whether or 
not the student response was a request for the correct answer,) 

It is estimated that about 3^000 such blocks of individual response 
data are collected for each student taking the course. 

The AID interpreter also collects data 'but the form is simpler since 
it contains no routines to analyze the student input. The AID interpreter 
collects the following information: 

1. Student number. 

2 . Date . 

3- Lesson identifier. (This is information sent to the AID inter- 
preter by the instructional program.) 

4. Problem number. (This also is sent by the instructional program.) 

5 . Subproblem number. (Again, this is sent by the instructional 
program.) 

6. Student input. (This is an exact character-by-character dupli- 
cate of everything typed by the student, excluding erasures.) 

It is estimated that about 300 such blocks of data are collected for 
each student taking the course, 

A variety of students have been enrolled in the AID course since the 
data collection routines were added to the programs. One of the largest 



groups is comprised of students in the "High Potential" program at the 
University of California at Los Angeles. These students are entering 
freshmen who do not meet the usual entrance requirements for UCLA but 
who, for one reason or another, are suspected to have a higher potential 
than revealed by their high school records or by entrance examinations. 
These students may be described as "culturally deprived," and are products 
of inner-city schools where average achievement is quite low. 

A second large group of students are from DeAnza College in Cupertino, 
California. DeAnza is a community college located a short distance from 
Stanford. The DeAnza students who have enrolled in the course were all 
unprovisionally admitted to DeAnza and most of them have a better high 
school background than the UCLA students. 

A number of NASA personnel, from the Ames Research Center at Moffett 
Field, California, and from the Manned Spacecraft Center in Houston, Texas, 
have also enrolled for the course. 

The Institute is providing CAI for hearing impaired students in 
several schools, and a half-dozen of these handicapped youngsters have 
also enrolled for the course during the last year. 

5» The Curriculum : Lessons 1 to 21 

The content of the course "Computer- assisted Instruction in Program- 
ming: AID" has been described elsewhere (Friend & Atkinson, 1971), so a 
complete description of the entire course will not be repeated here. The 
research reported in this paper concerns only the main strand Lessons 1 
to 21, and these lessons are described below in considerable detail. 

Lessons 1 to 21 contain three "blocks" of lessons; each of the blocks 
contains five tutorial lessons, one self- test and one general review: 
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Lessons 1 to 5 - Tutorial 

Lesson 6 - Self-test of Lessons 1 to 5 
Lesson 7 - General Review of Lessons 1 to 5 

Lessons 8 to 12 - Tutorial 

Lesson 13 - Self- test of Lessons 8 to 12 
Lesson lh - General Review of Lessons 8 to 12 

Lessons 15 to 19 - Tutorial 

Lesson 20 - Self-test of Lessons 15 to 19 
Lesson 21 - General Review of Lessons 15 to 19 

These lessons constitute a brief introduction to programming, cover- 
ing such concepts as stored programs, use of variables, fundamentals of 
input and output, the syntax of algebraic expressions and Boolean state- 
ments, definitions of functions, conditional clauses and branching, core 
and disk storage, use of subroutines, and some debugging techniques . 

Some of these concepts (input and output, core and disk storage, sub- 
routines) are discussed very briefly, while others (syntax of algebraic 
expression, syntax and meaning of Boolean statements) are covered more 
extensively. One major programming essential that is not introduced in 
the first 21 lessons is the loop, which is introduced in Lesson 23 • Lists, 
arrays, trigonometric and exponential functions, recursive functions and 
the truth function are also introduced in later lessons, A brief outline 
of Lessons 1 to 21 is given in Table 1. 

The 15 tutorial lessons in the first 21 lessons vary in length, de- 
pending upon the concepts covered by the lesson. Lesson 12, for example, 
covers two very simple commands and contains only 14 exercises, whereas 
Lesson 15 introduces Boolean statements and conditional clauses and 
contains 62 exercises. Also, the kinds of exercises in the lessons vary 
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Table 1 





Brief Outline: Lessons 1 to 21 


Lesson 1 


Using the Instructional Program 


Lesson 2 


Using AID for Arithmetic 


Lesson 3 


Order of Arithmetic Operations 


Lesson 4 


Exponents and Scientific Notation 


Lesson 5 


The SET and DELETE Commands 


Lesson 6 


Test of Lessons 1 to 5 


Lesson 7 


General Review of Lessons 1 to 5 


Lesson 8 


The LET Command 


Lesson 9 


Some Standard AID Functions 


Lesson 10 


Indirect Steps } the DO Command , the FOR Clause 


Lesson 11 


Parts 


Lesson 12 


The DEMAND Command 


Lesson 13 


Test of Lessons 8 to 12 


Lesson 14 


General Review of Lessons 8 to 12 


Lesson 15 


Relations and the Use of the IF Clause 


Lesson 16 


The TO Command 


Lesson 17 


Debugging Techniques 


Lesson 18 


The Indirect Use of the DO Command 


Lesson 19 


De bugging , Permanent Storage 


Lesson 20 


Test of Lessons 15 to 19 


Lesson 21 


General Review of lessons 15 to 19 



with the subject matter. Lesson 15 has 2b true-false exercises, reflec- 
ting the content of the lesson (Boolean statements). Lesson 8, in 
contrast, has no true- false exercises, instead it has 32 exercises that 
require the student to predict the result of using given AID commands. 
Table 2 shows the number and type of exercises in Lessons 1 to 21, ex- 
cluding the general review Lessons 7 ? 14 and 21. 

The exercises are categorized into the 13 different problem types 
listed in Table 2, The first four types are multiple-choice exercises. 

In the AID course, multiple-choice exercises may have more than one 
correct choice, and the student response is not. correct unless all cor- 
rect choices are listed. The multiple-choice exercises in each lesson 
are all classified according to the number of correct choices except for 
the few that include the choice, 

N. NONE OF THE ABOVE, 

and are classified separately. Of the 675 exercises in the first 2.1 
lessons there are 80 multiple- choice exercises, 56 of which have a 
single correct choice. 

There are a number of exercises that appear to be constructed- 
response exercises in that the student is not presented with a list of 
possible answers from which to choose. However, closer inspection re- 
veals that there are actually a limited number of choices of a form 
clearly implied in the statement of the problem. The following exercises 
for example, imply only two choices: 

Lesson 19, Exercise 8. 

SUPPOSE AID FOUND A SYNTAX ERROR IN STEP 17.2. DO YOU 
HAVE TO DELETE STEP 17 . 2 BEFORE YOU RETYPE IT? 
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Table 2 



Number of Exercises, by Type, in Lessons 1 to 21 
(Excluding General Reviews: Lessons 7, l4, 21) 



Lesson Number 


12 3 


4 


5 6 


8 


9 


10 


11 


12 


13 


15 


16 


17 


18 


19 


20 


s 

Total 


Multiple- Choice 
1 Correct Choice 


3 4 7 


6 


5 2 


1 


1 


2 






2 


5 


5 


1 


6 


1 


5 


56 


Multiple*' Choice 
2 Correct Choices 


4 1 




1 2 






1 


1 




1 












1 


12 


Multiple- Choice 
3 or Mo:te'\Correct 
Choices 


OJ 

OJ 

OJ 




1 




- 


1 




















8 


Multiple- Choice 
Correct Choice: NONE 


1 1 


1 


■ 1 


























4 


Total Multiple-Choice 
Exercises 


5 11 11 


7 


6 6 


1 


1 


4 


1 


0 


3 


5 


5 


1 


6 


I 


6 


80 


Yes-No Exercises 
(except opinion 
questions) 


2 




















1 






7 


1 


11 


True-False 






9 














24 










1 


34 


Other Implied- Choice 


1 


1 
















2 








1 




5 


Total Implied Choice 


ro 

f— * 

o 


1 


0 9 


0 


0 


0 


0 


0 


0 


26 


1 


0 


0 


8 


2 


50 


Predicted AID Response 


4 19 25 


5 12 


32 


17 


1 






13 


6 










1 


135 


Constructed AID Command 


2 7 


1 


9 5 


8 




2 


6 


1 


8 




1 






7 


6 


63 


Reported Result of AID 
Use 


3 3 


3 


13 2 


4 


6 


4 


4 


3 


1 


1 












47 


Other Constructed- 
Response Exercises 


3 1 15 


17 4 


8 




13 


5 


2 


3 


10 


10 


18 


22 


4 


10 


145 


Total Constructed 
Response # 


3 9 30 44 44 23 52 


23 


20 


15 


6 


25 


17 


11 


IB 


22 


11 


17 


390 


"Use AID ,f 


3 3 


’ 3 


9 2 


4 


3 


5 


6 


5 


2 


6 


2 




1 


4 




58 


Opinion (or preference) 


.5 4 5 


6 


4 3 


5 


4 


4 


8 


3 


3 


8 


8 


4 


6 


’ 8 


2 


90 


Unclassified Exercises 


3 






















4 








7 


Total 


18 28 49 61 63 43 62 31 


33 


so 30 


14 


33 


62 


27 


27 


35 


32 


27 


675 



Lesson 6, Exercise 8. 

ANSWER TRUE OR FALSE: 

CTRL-Z WILL STOP THE AID INTERPRETER. 

Lesson 4, Exercise 27. 

IF YOU USED THIS COMMAND 
TYPE 1/100 

WOULD AID GIVE THE ANSWER IN DECIMAL FORM OR IN 

SCIENTIFIC NOTATION? 

Exercises of this type are labeled "implied choice" exercises and are 
classified as yes-no exercises, true-false exercises, or "other implied- 
choice" exercises. In the first 21 lessons there are 50 such exercises. 

Three-hundred ninety-two of the exercises in the Table 2 lessons 
fall into the third major group comprising true constructed- response 
exercises. This group is subdivided into four classes; predicted AID 
responses, constructed AID commands, reported result of .AID use, and 
"other constructed-response" exercises. The predicted AID responses 
contain questions like "What would AID answer if you gave this command 
One-hundred and five of the constructed-response exercises are 
of this type, 

A smaller class of constructed response exercises includes the 63 
exercises that require the. student to construct a complete AID command. 
This class does not include all, or even most, of the AID commands that 
the student must construct. It includes exercises that pass the con- 
structed AID commands to the instructional program; excluded are exercises 
that require direct communication with the AID interpreter. The instruc- 
tional program is more capable than the AID interpreter of analyzing 
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constructed AID commands in detail and of giving meaningful messages to 
students who make errors when they first learn a new command. The usual 
sequence of instruction in introducing a new AID command is as follows : 

First, an example of the command is shown, and its use is 
explained. In this step, students are usually required to determine 
which of several forms of the command are syntactically correct. 

Second, the student is shown several examples and asked to 
determine what AID would respond if such a command were given to 
the AID interpreter. 

Third, the student is asked to construct commands that would 
result in a specified action. 

Fourth, the student is asked to start the AID interpreter and 
give commands of the new kind directly to AID. 

Fifth, after using the AID interpreter as directed, the 
student is requested to report on the results given him by AID. 
Exercises in this fifth step are classified in Table 2 as "reported 
result of AID use." In these exercises, the instructional program can 
infer the kind of errors a student is making and give him remedial in- 
struction if* needed. The device of asking students to switch to the 
interpreter for computation and to switch back to the teaching program 
to report results is one of the weakest features of the course, and it 
is used only from necessity. Clearly, an efficient means of interfacing 
the two programs is needed to pass information invisibly, and provide 
meaningful instruction for the students who need it. Only 47 of the 
390 constructed- response exercises are of the type needed in the fifth 
step, because an overall teaching strategy used in the course is to 
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encourage students to assume responsibility for ferreting out their own 
errors and for determining if they have a correct program. 

The fourth class of constructed-response exercises, called "other 
constructed- response exercises," contains 145 exercises of considerable 
variety. This class will be further subdivided in a later analysis. 

The miscellaneous exercises contain a class designated in Table 2 
as "use AID." These exercises require the student to use the AID inter- 
preter in solving a problem. The problems range in complexity from 
copying commands in order to observe their consequences to solving 
complex problems by writing and debugging complete programs. Many of 
these exercises contain three or four problems, and the student is asked 
to solve all of the stated problems before he switches back to the in- 
structional program and reports the results. 

Also in the miscellaneous group are those exercises that elicit an 
opinion or preference. These exercises either ask the student to express 
a preference for the sequence of instruction (Do you want a summary of 
this lesson?) or to give a self-evaluation of his competence (Do you 
remember how to start the AID interpreter?). The exercises are fairly 
evenly distributed over lessons and there are 90 of them in the first 
21 lessons. 

In the following paragraphs, each of the first 21 lessons is de- 
scribed and characteristic exercises from the lesson are given. 

Lesson 1: Using the Instructional Program 

Lesson 1 is a set of 18 short exercises explaining how to use the 
instructional program. The mechanics of typing and erasing responses 
are explained, and instructions are given for starting and stopping the 
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program. The student is taught how to use optional control keys to get 
additional instruction (HINT and TELL commands) or to alter the sequence 
of exercises (the GO command). 

The style of instruction in Lesson 1, as in succeeding lessons, is 
informal, and it does not always give explicit directions, requiring the 
student to attend the instructions carefully and learn by induction from 
the examples given. 

Two of the exercises in Lesson 1 are given here. 

Lesson 1, Exercise 3: 



IF MULTIPLE CHOICE PROBLEMS HAVE MORE THAN ONE CORRECT 
ANSWER, YOU CAN LIST THE CORRECT CHOICES IN ANY ORDER. 
SUPPOSE B, C AND D ARE THE CORRECT CHOICES FOR A PROBLEM. 
WHICH OF THESE WOULD BE CORRECT WAYS TO ANSWER? 

A. D, B, C, A 

B. B, D, C 

C. B, C, D 

D. D, B, C 



Exercise 14 illustrates an instructional strategy that is used in 
many lessons. At the end of a lesson, its content is briefly summarized, 
and the student is given the option of reviewing topics he is unsure of. 
If the student responds affirmatively, he is given th*= review associated 
with that lesson. In this way, the student is made responsible for the 



Lesson 1, Exercise 14: 



FROM LESSON 1, YOU SHOULD HAVE LEARNED HOW TO SIGN ON 
AND OFF, HOW TO START AND STOP THE TEACHING PROGRAM, 
HOW TO GET A HINT, AND HOW TO USE CTRL-G. DO YOU WANT 
TO REVIEW ANY OF THESE TOPICS? 
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material covered in each lesson and is forced to judge if he is competent 
to proceed with the course or if he needs additional instruction and 
practice. 

In Lesson 1, five of the 18 exercises are multiple choice, similar 
in form to Exercise 3 shown above. This proportion of multiple-choice 
exercises is fairly typical of the course. The multiple -choice format 
is chosen over a constructed response format depending on the objective 
of the exercise. For example, if the purpose is to teach students to 
discriminate between syntactically correct and incorrect AID commands, 
a multiple- choice exercise is used; if the purpose is to teach the con- 
struction of an AID command, a constructed response will be requested. 

Of the 13 exercises remaining in Lesson 1, seven could appropriately 
be labeled "implied multiple choice" since the statement of the exercise 
clearly implies only a small number of possible responses. In order to 
keep the terminology straight, however, exercises that are not in the 
conventional multiple -choice format, will be referred to as "implied 
choice" exercises, and the term, "multiple choice," will be reserved for 
exercises that list and label a set of choices and require that the 
student respond by typing the label or labels. 

One of the most frequently used implied- choice exercises is the 
yes-no exercise. Many of these are designed to elicit opinion rather 
than information and have no "correct" answer. In Lesson 1, five of 
the seven implied- choice exercises are of this type and in Table 2 are 
classified as "opinion exercises" rather than "implied-choice exercises." 
This proportion is greater than that of later lessons, but still, in- 
dicates the style of the lessons. 
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Lesson 2: Using AID for Arithmetic 



The 28 exercises in Lesson 2 teach the student how to start and 
stop the AID interpreter and how to use it for simple arithmetic by- 
employing the "TYPE" command. The AID symbols for the four simple arith- 
metic operations (+, -, *, and /) are taught, and the use of parentheses 
in arithmetic expressions is introduced. '£*y the end of the lesson, the 
student should be able to start, the AID interpreter and give simple, 
direct commands such as: 

TYPE 5/25 

TYPE 3.25 + 17.4 - 3.12 

TYPE 15 * 17 + 25 * 19 

One of the most persistent errors made by students learning any 
algebraic programming language is the omission of the multiplication 
operator in algebraic expressions. The source of this difficulty is the 
convention of using juxtaposition to indicate multiplication. For example, 
we ordinarily write 

a(b+c) 

without an explicit multiplication operator, but AID, like other algebraic 
programming languages, demands that the multiplication be indicated ex- 
plicitly. AID has an asterisk as the symbol for multiplication, as in 

A * (BtC) • 

In Lesson 2, there are a number of exercises aimed specifically at over- 
coming this error. 

The following are exercises from Lesson 2. 

Lesson 2, Exercise 15: 

WHAT WOULD AID ANSWER TO THIS COMMAND? 

TYPE 72/12 . . 



Lesson 2 , Exercise 19 : 

USE AID TO DO THESE PROBLEMS: 

1. FIND THE AREA OF A RECTANGLE WITH WIDTH 1,72375 AND 
LENGTH I 2 .OOI 325 . 

2. SUPPOSE A SQUARE OF WIDTH ,637825 IS CUT FROM THE 
ABOVE RECTANGLE. FIND THE AREA OF THE SQUARE. 

3. FIND THE AREA OF THE REMAINING PART OF THE RECTANGLE'. 

Of the 28 exercises in Lesson 2, .11 are multiple choice, five are 
imp lie d-mult i.p le choice, and 12 are true constricted response exercises. 
The constructed response exercises vary in difficulty, from the simple 
problem given in Exercise 15, above, to the problem given in Exercise 19 . 
Lesson 3 ; Order of Arithmetic Operations 

The arithmetic used in Lesson 2 was relatively simple, but in Lesson 
3 the complexity increases with the addition of the concept of hierarchy 
of operations and the use of parentheses. Because each AID command must 
be typed entirely on a single line, horizontal division bars for grouping 
cannot be used. Thus, an expression like 

JSL 

z+3 

using more than one line of type, must be translated into the AID 
expression 

X * Y / (Zd-3) 

with parentheses to show grouping. The AID expression is more difficult 
to construct, since it requires a conscious decision about the desired 
order of evaluation. 

Lesson 3 teaches the student how to force an order of evaluation by 
using parentheses. To do this, the student must recognize the difference 

!>. 
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between expressions like (l6 - 4) - 3 and 16 - (4-3)* After a few 
exercises on parentheses, rules are given for the: hierarchy of the four 
basic arithmetic operators ( + , -, *, and /), and a number of exercises 
are given in which the task is to determine the order of evaluation .if 
no parentheses are used. 

The following are two examples from Lesson 3- 
Lesson 3> Exercise 5: 

WHAT WILL AID ANSWER TO THIS COMMAND? 

TYPE .1/(100/10) 

Lesson 3> Exercise 10: 

LOCK AT THESE THREE COMMANDS. AID WILL GIVE THE SAME 

ANSWER TO TWO OF THEM. WHICH TWO? 

A. TYPE 3 + (2*4) 

B. TYPE (3+2) * 4 

C. TYPE 3+2*4 

Since Lesson 3 is primarily a review of algebraic notions that may 
be better understood by some students than by others, it provides more 
opportunity for individualized branching. A good student can complete 
Lesson 3 i n 27 exercises, but a poor student will be given additional 
practice and may do up to 49 exercises. 

The exercises in Lesson 3 are of medium difficulty, and most can 
be done quickly. Eleven of the exercises are multiple choice, and an 
additional five are implied- choice exercises. Three of the exercises, 
including one with three parts, require the student to use the AID in- 
terpreter. Nineteen of the exercises are similar to Exercise 5 above. 



Lesson 4: Exponents and Scientific Notation 



Lesson 4 is longer than average and extends the work on arithmetic 
expressions to include exponentiation. First, the concept of exponenti- 
ation is reviewed, with the introduction of the AID symbol (t). The rules 
for the hierarchy of operations are extended to include exponentiation, 
and the AID form of scientific notation is introduced. Negative exponents, 
fractional exponents, and zero as an exponent are also covered. Lesson 4, 
like Lesson 3.- is largely review of algebraic principles and may have 
been forgotten. The exercises also provide practice in reading and con- 
structing expressions in the form required by the AID interpreter. 

Some examples from Lesson 4 follow. 

Lesson 4, Exercise 6: 



Lesson 4 contains 6l exercises, of which 48 require constructed 
responses. Most of the constructed responses require arithmetic calcu- 
lations to answer questions like those illustrated in Exercise 6 above. 
Three exercises require the student to use the AID interpreter, and he 
is encouraged to use AID throughout the lesson. 

Lesson 5 ; The SET and DELETE Commands 

After the sizeable dose of arithmetic given in Lessons 3 and 4, 
Lesson 5 provides relief by returning to the mainstream of instruction 



WHAT IS THE VALUE OF 5*2/2? 



Lesson 4, Exercise 12: 



USE AID TO EVALUATE EACH OF THE FOLLOWING. 



1. 4 SQUARED TIMES 3.l4l6 

2. THE SUM OF 4 CUBED AND 6. 

3. THE SUM OF THE SQUARES OF 1, 2, 3 , 4, 5, 6, 7 AND 8 
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with the introduction of two new AID commands : the SET command and the 
DELETE command. SET is used in AID to assign values to real variables; 
DELETE is used to delete a previous assignment or definition. In AID, 
variables are single letters, and the SET and DELETE commands are easily- 
learned by most students. A number of word problems are given to illus- 
trate the use of the new commands. Lesson 5 also introduces the multiple- 
argument form of the TYPE command in which several TYPE commands can be 
combined into one by separating the arguments with commas (TYPE X,Y,X+Y). 

The following exercises are from Lesson 5* 

Lesson 5, Exercise 3? 

WHAT WILL AID ANSWER AFTER THESE COMMANDS? 

SET B = 1*5 
TYPE 3*B 




Lesson 5<> Exercise 31: 

TO FIND THE NEW AMOUNT IN A SAVINGS ACCOUNT, CALCULATE THE 
INTEREST AND ADD IT TO THE LAST BALANCE. START AID AND 
CALCULATE THE INTEREST AND THE NEW BALANCE .AFTER ONE YEAR 
FOR AN ACCOUNT WITH AN INTEREST RATE OF 4.5 PERCENT PER 
'YEAR AND A PREVIOUS BALANCE OF $3274.86. (ASK FOR A HINT 
IF YOU NEED ONE.) 

WHAT IS THE INTEREST ON THE ABOVE ACCOUNT TO THE NEAREST 
PENNY? 

WHAT IS THE NEW BALANCE IN THE ACCOUNT? 



Lesson 5, with its 63 exercises, is fairly long and has nine exer- 
cises that ask the student to use AID to solve problems. The most, 
difficult of these is shown in Exercise 31 above. The trend toward a 
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higher proportion of constructed responses continues here, with Lesson 5 
having only six multiple-choice problems. There are four implied- choice 
exercises, but all of these are requests for opinions from the student. 
Lesson 6; Test of Lessons 1 to $ 

Lesson 5 concludes the first five-lesson tutorial block and is fol- 
lowed by a self-test in Lesson 6 and a general review in Lesson 7 . Both 
Lessons 6 and 7 are optional. Lesson 6 contains 40 test questions and 
problems covering the material in Lessons 1 to 5* Like other self-tests, 
Lesson 6 supplies no hints, and students are allowed only one try on each 
exercise. However, the student may request the correct answers at any 
time by typing CTRL-T. Whenever a student misses an exercise, he is 
given a review reference and advised to review that topic before pro- 
ceeding with the course. 

The exercises in Lesson 6 are classified according to which lesson 



they are testing: 




Lesson Number 


Exercises in Lesson 6 Testing Given Lesson 


1 


2 , 3, 4, 5 , 6 , 7 , 9, 10 , 12 , 13, 14, 15, 16 


2 


8 , 11 , 17 , 18 , 19, 20 , 38 - 1 , 39-1 


3 


21, 22, 23 


4 


24, 25 , 26 , 27 


5 


28 , 29 , 30 , 31, 32 , 33, 34, 35, 36 , 37 



Exercises 1, 1-1 and 40 are "opinion" exercises and are not listed. 
Exercises 38 and 39 are "use AID" exercises and are not listed. Of the 
38 exercises in the list, 23 are constructed-response exercises. 
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Lesson J: General Review of Lessons 1 to 5 



The student is allowed to skip Lesson J r but he is advised to take 
it if he missed more than five problems in the Lesson 6 self- test. Les- 
son 7 uses a more complex branching scheme to allow students to review 
only selected portions of the preceding lessons. 

The following is an example from Lesson J . 

Lesson 7, Exercise 5 '« 

LESSON 4 WAS ABOUT EXPONENTS AND SCIENTIFIC NOTATION. 
FRACTIONAL EXPONENTS AND NEGATIVE EXPONENTS WERE 
DISCUSSED, AND ALSO THE USE OF 0 AND 1 AS EXPONENTS, 

THE ORDER OF ARITHMETIC OPERATIONS + - * / and °jo WAS 
COVERED. 

DO YOU WANT TO REVIEW ANY OF THESE THINGS? 

If a student answers "yes," he is sent to the review lesson for Lesson 4 
where he may review any of the lesson topics in any order he wants. 

This first general review also reminds students that they can con- 
trol the sequence of instruction by using the CTRL-G key and that the 
student manual provides an outline of the course. 

Lesson 8: The LET Command 

Lesson 8 is the beginning of a new lesson block, and introduces the 
powerful LET command, used in AID to define functions. The difference 
between LET and SET commands is discussed, and a variety of algebra 
problems is given. A major emphasis in Lesson 8 is on substituting 
arithmetic expressions for variables in algebraic expressions. Lesson 8 
is the first lesson that has optional "extra-credit" problems. 

The following are two exercises from Lesson 8. 




vV 



30 



Lesson 8, Exercise 4; 



WHAT WILL AID ANSWER? 



LET P(M) = Mt2 
TYPE P(6/2) 



Lesson 8, Exercise 28: 



USE AID TO DO THIS PROBLEM. DEFINE A FUNCTION TO CONVERT 
DEGREES FAHRENHEIT TO DEGREES CENTIGRADE, THEN CONVERT 
THESE TEMPERATURES TO CENTIGRADE: 



0 ., 10 , 32 , ? 2 , 212 



Of the 62 exercises in Lesson 8, over half are similar to Exercise 
4 above. There are four multi-part, problems that require the use of the 
AID interpreter. There is only one multiple-choice exercise in Lesson 8 
and none of the exercises are more difficult than Exercise 28 above. 

Lesson 9‘ Some Standard AID Func tions 

In Lesson 9, the student is introduced to four functions already 
defined in AID, These are: 



These functions, together with those defined by the student, are 
used in several problems requiring the use of the .AID interpreter. 

The following are two exercises from Lesson 9» 

Lesson 9, Exercise 2: 



SQRT(X) « the square root function, 



IP(X) - the "integer part 5 ' function, 
FP(X) - the "fraction part" functi 
SGN(X) - the sign function. 



WHAT WILL AID ANSWER? 



TYPE 3 * SQRT(IOO) 
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Lesson 9 , Exercise l4: 

YOU CAN USE THE AID FUNCTION FP(x) TO FIND OUT IF ONE 
NUMBER CAN BE DIVIDED BY ANOTHER WITHOUT A REMAINDER.... 

IS 2976 EVENLY DIVISIBLE BY 3? 

Lesson 9 has 31 exercises of which 17 are similar to Exercise 2 
above. 

Lesson 10: Indirect Steps, the DO Command, the FOR Clause 

In Lesson 10, the concept of a stored program is introduced. Up to 
this point, students have been using AID as a desk calculator, doing all 
exercises with direct commands, i.e., commands that are executed immed- 
iately. In this lesson, the students are taught that TYPE commands can 
be stored for later execution by prefacing the command with a step number, 
as in the following examples : 

2.1 TYPE F(l6) 

4.7 TYPE Xt2,Xt3 . 

They are also taught how to execute these stored commands by using 
a DO command. Two variants of the FOR clause are used to modify DO com- 
mands. In the first variant, values for the iteration variable are given 
by a simple listing: 

DO STEP 17.3 FOR Y = 1,2,7,14.3* 

In the. second variant, values for the iteration variable are given in a 
range specification that specifies an initial value for the variable, a 
step siae, and a final value: 

DO STEP 5 FOR X = 4(2)9. 

This command specifies that X will assume the value 3> then be incremented 
by 2 after each iteration of step 5 until X > 9* This is equivalent to 
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the FORTRAN form, 

DO 5 X = 4,9,2 
5 < statement 5 >> 

and the ALGOL form 

FOR X - 4 STEP 2 UNTIL 9 DO 
< statement 5 >• 

The following are some examples from Lesson 10. 

Lesson 10, Exercise 12: 

USING AID, WRITE AN INDIRECT STEP THAT WILL CONVERT MILES 
PER HOUR TO FEET PER SECOND. THEN CONVERT ALL OF THESE 
TO FEET PER SECOND: 

10 MILES PER HOUR 
100 MILES PER HOUR 
65 MILES PER HOUR 
1023 MILES PER HOUR 



Lesson 10, Exercise 17 : 

WHAT VALUES OF A. WILL BE USED IF THIS COMMAND IS GIVEN? 
DO STEP 73-7 FOR A = 5(10)35 

A. 5, 10, 15, 20, 25, 30, 35 

B. 10, 15, 20, 25, 30, 35 

C. 5 , 15 , 25, 35 



Of the 33 exercises in Lesson 10, four are multiple choice and an- 
other four are implied choice. Most of the constructed- response exercises 
are quite simple, and Exercise 12 above is the only one that requires any 
problem-solving skills . 

Lesson 11: Parts 

Lesson 11 explains how indirect (stored) steps are grouped into 
"parts." Steps 12.1, 12. 7 , and 12.8, for example, are grouped as 
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"Part 12 , " and can be executed by a single command: 

DO PART 12. 

The sequence of execution depends only upon the numerical order of the 
step numbers, and not upon the sequence in which they were written. Thus, 
steps 3.75, 3.2 and 3*8 will be executed in the order: 3-2, 3«75> 3-8, 

This concept is clear to most students. The only difficulty is caused 
by step numbers with trailing zeros; some students fail to order cor- 
rectly a sequence like 3*5* 3*8, 3*10 (the correct order is 3«10, 3-5* 



Here are two examples from Lesson 11. 

Lesson 11, Exercise 5: 

YOU CAN TYPE THE STEPS IN ANY ORDER, BUT AID WILL ALWAYS 
DO THEM IN NUMERICAL ORDER. WHICH STEP WILL BE DONE FIRST? 

17.4 TYPE XtY 

17.5 SET N=5 

17.2 SET X=10 

17.3 SET Y=2 

Lesson 11, Exercise 11: 

A PART (SET OF INDIRECT STEPS) IS ALSO CALLED A PROGRAM. 

USE AID TO WRITE A PROGRAM THAT WILL LIST THE RADIUS, 
DIAMETER, CIRCUMFERENCE, AND AREA OF A CIRCLE OF RADIUS 
R. THEN USE THE PROGRAM FOR R = 10 , 20, 30 , 40 AND 50. 

Lesson 11 has 30 exercises, with an unusually high proportion of 
"opinion" questions (8 out of 30). Only one of the 30 exercises is 
multiple choice, and six of the 30 require the student to use the AID 
interpreter, a slightly higher proportion than was found in earlier 
lessons . 



3.8). 
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Lesson 12: The DEMAND Command 



In Lesson 12 the DEMAND command is introduced. The DEMAND command 
is used in AID programs for teletype input. The DEMAND command, unlike 
previously introduced commands, can be used only indirectly, as a stored 
command. This lesson also introduces a variant of the DO command: 

DO PART 7, 5 TIMES 

The following are exercises from Lesson .12. 

Lesson 12, Exercise 4: 

START AID AND WRITE A PROGRAM THAT WILL ASK YOU FOR 3 
NUMBERS, A, B, AND C, AND THEN GIVE YOU THE AVERAGE OF 
THE 3 NUMBERS. AFTER YOU HAVE TESTED YOUR PROGRAM, USE 
IT TO FIND THE AVERAGE OF 

A = 179-053 
B = 23.7 
c = 271.0015 

Lesson 12, Exercise 5; 

WHAT COMMAND WOULD YOU USE IF YOU WANTED PART 2 DONE 
7 TIMES? 

Lesson 12 is very short, containing only 14 exercises. None is 
multiple choice, and five require the student to use the AID interpreter. 
Lesson 13: Test of Lessons 8 to 12 

Lessons 8 to 12 constitute the second five-lesson tutorial block of 
instruction and are followed by an optional self-test (Lesson 13) and a 
review (Lesson 14). Lesson 13 is structured like other self-tests; the 
student is given only one try for each exercise, and there are no hints 
provided. A student who cannot do an exercise can request the correct 
answer by typing CTRL-T. 
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The following classifies the exercises in Lesson 13 according to 



which lesson is being tested. 

Lesson Number Exercises in Lesson 13 Testing Given Lesson 

8 1 - 1 , 2 , 3 , k, 5 , 6 , 7 , 8 

9 9 , 10, 11, 12, 13, 14 , 16 

10 17, 18, 19, 20, 21, 22, 23, 24 

11 25, 26, 27 

12 28, 29-1 

Five of the 33 exercises in Lesson 13 are not included in the list. Ex- 
ercises 1 , 15-1 and 30 are "opinion" questions, and Exercises 15 and 29 
are "use AID" exercises. Of the 28 exercises in the list, 25 are con- 
structed- response exercises. 

Lesson 14 : General Review of Lessons 8 to 12 

Lesson 14 , the general review of Lessons 8 to 12 , is optional but 
is recommended for students who missed more than three problems in the 
preceding self- test. Here is one example from Lesson 14 . 

Lesson 14 , Exercise 3 : 

LESSON 8 WAS ABOUT THE "LET" COMMAND AND HOW TO USE IT 
TO DEFINE A FUNCTION. FUNCTIONS OF 2 AND 3 VARIABLES 
WERE DISCUSSED. INSTRUCTIONS FOR PRINTING AND DELETING 
A FUNCTION WERE GIVEN. 

DO YOU WANT TO REVIEW ANY OF LESSON 8? 

The structure of Lesson 14 , like that of other general reviews, 
allows a student who answers "yes' 1 to branch to the review for the les- 
son and to review any of the lessor, topics in any order. 
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Lesson 15 : Relations and the Use of the IF Clause 



Lesson 15 begins a new lesson block and introduces the most powerful 
programming tool: the conditional clause. The conditional (IF) clause 
may be appended to any of the commands so far introduced. The following 
AID symbols for arithmetic relations are introduced: 



The terms "positive , " "negative," and "non- negative" are reviewed. The 
Boolean connectives "and" and "or" are also introduced. Students are 
required to write several programs using conditional branching. 

The following are examples from Lesson 15. 

Lesson 15, Exercise l4: 



STUDY THIS PROGRAM. 

49.5 TYPE X IF X > Y 

49.6 TYPE Y IF X < = Y 
DO PART 49. 

IF X = 12,1 AND Y = 6, WHAT WILL AID ANSWER? 



WRITE A PROGRAM THAT WILL PRINT "SAME" IF ALL THREE 
NUMBERS X, Y, AND Z HAVE THE SAME SIGN. THE PROGRAM 
SHOULD PRINT "DIFFERENT" IF THE NUMBERS DO NOT ALL 
HAVE THE SAME SIGN, 



Of the 62 exercises in Lesson 15 , a large number (24) are true-false 
exercises. The students are required to use AID in six exercises similar 
to Exercise 20 above. 



< less than, 

> greater than, 

<= less than or equal, 

>= greater than or equal, 

# not equal. 



Lesson 15, Exercise 20: 
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Lesson 16: The TO Command 



Lesson 1 6 reviews the use of conditionals and introduces conditional 
branching. 

The following are two examples from Lesson 16. 

Lesson l6, Exercise 3: 

HERE IS A PROGRAM THAT CALCULATES THE AREA OF A RECTANGLE 
OF LENGTH L AND WIDTH W. IF EITHER L OR W IS NEGATIVE, 

PART 15 IS USED TO GIVE AN "ERROR" MESSAGE. 

14.1 DEMAND L 

14.2 TO PART 15 IF L < 0 

14.3 DEMAND W 

14.4 TO PART 15 IF W < 0 

14.5 TYPE L * W 

15.1 TYPE "DO NOT USE NEGATIVE NUMBERS." 

WHICH STEPS WILL BE DONE IFL = 5ANDW = -3? 

Lesson 1 6 , Exercise 4: 

WRITE A PROGRAM THAT WILL DEMAND A RADIUS R AND THEN 
CALCULATE THE AREA OF A CIRCLE WITH THAT RADIUS. USE 
TWO PARTS, ONE FOR THE MAIN PROGRAM AND ONE FOR AN 
"ERROR" ROUTINE TO BE USED IF R IS NEGATIVE. 

There are 27 exercises in Lesson 16, with five multiple choice and 
eight "opinion" questions. 

Lesson 17: Debugging Techniques 

Lesson 17 concentrates on debugging techniques, showing the student 
how to trace a program by making a table listing the steps in order of 
execution. 

Here is one example from Lesson 17, Exercise 3: 



FOR PRACTICE, LET'S MAKE A TRACE OF THIS PROGRAM, ASSUMING 
A = 3. 

31.3 DEMAND A 

31.2 SET B = At2 - 10 

31.3 SET C = A IF A > B 

31.4 SET C = B IF A < = B 

31.5 TYPE B 

31.6 TYPE C 



FILL IN THE VALUES OF C 


IN THIS 


TRACE (STARTING AT STEP 31.3). 


STEP 


A 


B 


c 


31.1 


3 


- 


- 


31.2 


3 


-1 


- 


31-3 


3 


-1 


? 


31.4 


3 


-1 


? 


31.5 


3 


-1 


? 


31.6 


3 


-1 


? 



Of the 27 exercises in Lesson 17, 18 are similar to the exercise 
above. Four exercises require the student to write a complete trace 
with paper and pencil and check his answer by typing CTRL-T. 

Lesson 18: The Indirect Use of the DO Command 

In Lesson 18, the indirect use of the DO command is introduced. Up 
to this point, the student has been using DO commands directly to execute 
programs or single steps. The DO command can also be given indirectly to 
execute subroutines . A conditional clause is frequently used with in- 
direct DO commands. 

The following is an exercise from Lesson 18, 
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Lesson 18, Exercise 2: 

WHEN AID COMES TO AN INDIRECT "DO” COMMAND, IT WILL DO 
THE STEP OR PART INDICATED AND THEN RETURN TO THE STEP 
AFTER THE ”DO" COMMAND. 

16.1 DO STEP 2.1 IF Q. < 0 

16.2 TYPE Q 

2.1 SET Q = - Q 
DO PART 16 

IF Q = 3; THE STEPS WILL BE DONE IN WHICH ORDER? 



A 


B 


C 


D 




— 


— 




16.1 


16 . i 


16.1 


16.1 


2.1 

16.2 


16.2 

2.1 


2.1 


16.2 



There are 35 exercises in Lesson 18, with only one programming 
problem requiring the use of AID. 

L esson 19: D e bugging, Permanent Storag e 

The first half of Lesson 19 is an optional section of tips for 
writing and debugging programs. This section is primarily for students 
who have; been having difficulty with the programming problems in pre- 
ceding lessons. The second half of the lesson describes the: difference 
between core memory and disk storage and teaches the students how to 
store programs on the disk by using the AID file commands’ USE, FILE, 
RECALL , and DISCARD. 

Here are two examples from Lesson 19 . 

Lesson 19, Exercise 13: 

WHAT COMMAND WOULD YOU USE TO FILE PART 29 AS ITEM 3? 
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Lesson 1 9, Exercise 17 : 

IF A NUMBER B WAS FILED AS ITEM 6, YOU WOULD RECALL IT 
BY TYPING 

USE FILE 100 
AND THEN WHAT? 

Of the 32 exercises in Lesson 19, eight are requests for opinions, 
and four ask the student to use AID. 

Lesson 20: Test of Lessons 15 to 19 

Lesson 19 completes the third lesson block. Lesson 20 is a test of 
Lessons 15 to 19, and is structured like the tests in Lessons 6 and 13. 
Lesson 20 contains 27 exercises, of which two are requests for opinions. 
The other 25 can be grouped according to which lessons they test: 

Lesson Exercises in Lesson 20 Testing Given Lesson 

15 1-1, 2, 3, 4, 5, 6 

16 7, 16 

17 8, 9, 10 

18 11, 12, 13, 14, 15 

19 17, 18, 19, 20, 21, 22, 23, 24, 25 

There are six multiple- choice exercises, two "imp lied- choice" exer- 
cises, and 17 constructed responses. 

Lesson 21; General Review of Lessons 15 to 19 

Lesson 21 is a general review of the lessons tested by Lesson 20. 

Like other general reviews, it is optional and is recommended for students 
who missed more than three problems in the test. 

Here is one example from Lesson 21: 



Lesson 21 , Exercise 8: 

LESSON 19 EXPLAINED HOW TO PLAN, WRITE, AND EDIT A PROGRAM; 

WHAT KINDS OF ERRORS THERE ARE AND HOW TO CORRECT THEM; AND 

HOW TO USE PERMANENT STORAGE. 

DO YOU WANT TO REVIEW LESSON 19? 

A student who responds "yes" will be given the review lesson for 
Lesson 19 . 

6. The Daily Report 

A daily report program was provided to inform teachers of the pro- 
gress of individual students. The report lists the students in given 
classes, shows their current position in the curriculum, and indicates 
if they used the curriculum on the day of the report. The position of 
each student is indicated by printing the number of the last problem he 
completed in the tutorial lessons, in the reviews, in the summaries, and 
in the extra-credit problems. Ey comparison with previous daily reports, 
the teacher (or researcher) can judge about how much an individual student 
has progressed, and he can compare the positions of different students. 

However, the daily report provides only a rough measure of progress 
since the students themselves control the sequence of instruction. A 
student may be on Lesson 5 one day and on Lesson 12 the next, either by 
diligently working through all the intervening lessons or by simply 
skipping directly to Lesson 12. Also, a student may decide to go back 
to review a previous lesson, and his daily report will show that he was 
on Lesson 12 one day and had regressed to Lesson 5 the next. Even if 
one assumes that students are working their way straight through the 
course, it is hard to get a precise measure of progress from the reports 



because of the varying lengths of lessons and the uneven dispersion of 
time-consuming programming problems. Nevertheless, the daily reports 
can provide an adequate indication of average rate of progress through 
the course and of variation in the rates of progress among students. 

To illustrate use of the daily reports, one class of 39 students 
was selected as a sample. All these students were enrolled in the UCLA 
"High Potential" program. The daily report information for the class is 
summarized for each student and presented in Table 3 . Number of days 
worked, number of lessons completed, and rate of progress in number of 
lessons per day are listed in Table 3 . This particular class was chosen 
because of its large enrollment, but it is atypical because attendance 
for AID was voluntary. Of 48 possible workdays, the average number of 
days worked by the 39 students is 10.4 with a range from 1 to 36 days. 

This average is high for strictly voluntary attendance. Average number 
of lessons completed is 14.1 with a range from 2 to 36 lessons, indicating 
that the students completed slightly less than one-third of the course In 
the time allotted. 

To illustrate the individuality of the students, progress in number 
of lessons completed for three students (Nos. 2, 7, and 1,1) from the 
class tabulated in Table 3 is graphed in Figure 3? Student 2 can be 
characterized as slow and steady, Student 7 can be characterized as fast 
and steady, and Student 11 can be characterized as persistent but erratic. 

7 . Item Analysis of INST Data 

In this preliminary report, only data from first responses for Les- 
sons 1 to 21 are analyzed. The data used were collected by the INST 







Table 3 



Daily Report Summary for One Class of 39 Participating Students 



Student 


Number of 
days worked 


Number of lessons 
completed 


Number of lessons 
completed per day 


1 


22 


23 


l.o 4 


2 


17 


11 


.65 


3 


17 


27 


1.59 


4 


2 


5 


2.50 


5 


13 


18 


1.38 


6 


13 


8 


.62 


7 


15 


24 


1.6o 


8 


17 


26 


1.53 


9 


1 


9 


9.00 


10 


1 


2 


2.00 


11 


36 


36 


1.00 


12 


6 


4 


.67 


13 


3 


2 


.67 


14 


2 


4 


2.00 


15 


6 


5 


.83 


16 


2 


3 


1.50 


17 


8 


20 


2.50 


18 


13 


13 


1.00 


19 


16 


22 


1.38 


20 


9 


9 


1.00 


21 


4 


8 


2.00 


22 


19 


22 


1.16 


23 


20 


31 


1-55 


24 


18 


22 


1.22 


25 


8 


20 


2.50 


26 


11 


11 


1.00 


27 


5 


8 


1.60 
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Student 



Number of 
days worked 



Table 3 (cont'd) 



Number of lessons 
completed 



Number of lessons 
completed per day 



28 


1 


2 


2.00 


29 


6 


8 


1-33 


30 


8 


11 


1.38 


31 


2 


4 


2.00 


32 


17 


21 


1.24 


33 


10 


23 


2.30 


34 


18 


17 


.94 


35 


7 


15 


2.14 


36 


6 


8 


1.33 


37 


20 


20 


1.00 


38 


1 


7 


7.00 


39 


_5 


21 


4.20 


Mean 


10.38 


14.10 


1.86 


S.D. 


7.80 


9.02 


1.62 


N 


39 


39 


39 
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Figure 3- Student progress in number of lessons completed 
after each day of work in AID for three students. 
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instructional program. No data collected by the AID interpreter are 
included in this section. 

Because of the unique branching structure used in the general review, 
Lessons 7 , 14 and 21 are not included in the analysis. The self-test 
Lessons 6 , 13 and 20 are included for comparison. 

To describe the statistics reported, it is necessary to explicate 
the definitions of "first response" and of "correct response" used by the 
data collection and analysis routines. "First response" is defined to be 
the first of a set of contiguous responses made in any one session to a 
single exercise, with the following two exceptions. First, a student may, 
through his own volition or by automatic action of the program, repeat an 
exercise. In such a case, the first response made by the student the 
second time he encounters the exercise is counted again as a first re- 
sponse to that exercise. Thus a total of 35 first responses to a given 
exercise may be the work of only 3^- students. Second, if a student 
terminates a session after responding incorrectly to an exercise, he will 
commence the next session with that same exercise; his first response in 
the subsequent session will be tallied as another first response to the 
exercise. In actual practice, the effect of these "extra" first responses 
is negligible. 

Some responses are not considered at all in the tally of first re- 
sponses. These are responses that cannot be classified as correct or 
incorrect. The only responses that fall into this category are question 
mark (student request for a hint), CTRL-G (student request for a change 
of problem sequence), CTRL-A (student request for a repeat of the problem), 
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and CTRL-Z (student request for sign-off). If any of these occur as a 
first response, they are ignored. 

Excluding the lessons and exercises not graded, 6,512 first .responses 
made by the 68 students who completed one or more AID lessons during the 
first semester of the 1972-73 school year were analyzed for this report. 

Of the 68 students, 39 were drawn from the UCLA "High Potential" program, 

26 from Ames -NASA, and three from the network of schools for the deaf. 

The other critical definition used is that of "correct response." 

In general, the definition of "correct" is supplied by the programming 
and is what one would expect. If the correct answer to an exercise is 
"true," the student response may be "true" or "t," and any other response 
such as "false" or "help" or "yes" is classified as incorrect. 

Some exercises have no clearly defined correct answer, however. 

First, requests for sequencing, "Do you want to try the extra-credit prob- 
lems for this lesson?’’, cannot be said to have a right or wrong answer. 
Second, the programming of a few exercises precluded easy classification ; 
there were seven such exercises in the 675 exercises considered. Third, 
some exercises ask the student to use AID to solve a stated problem. We 
could consider the response to be correct if the student did indeed use 
AID, but on a less superficial level we need some analysis of the student*s 
use of the AID interpreter. This implies a routine that can judge the 
correctness of a student-written program. Unfortunately, such a routine 
is not available. The general solution to the problem of proving the 
correctness of a computer program has been shown to be recursively un~ 
solvable (Davis, 1958). While many particular cases of this problem are 
solvable, it remains a deep and non-trivial problem to constrict an 
algorithm that will prove correctness for a comprehensive set of student 










solutions. Some data collected by the AID interpreter have been hand- 
graded and will be described later in this report. 

Problem Types 

A summary of correct first responses by problem type is shown in 
Table 4. Number correct, number of first responses, and percent correct 
are shown for each problem type. 

This summary reveals several interesting results from the item 
analysis. First, the proportion correct for all exercises is only 65»7 $, 
a figure at least 10$ lower than predicted after developmental testing 
of the program. This result is probably explained by the fact that most 
of the data for this analysis was obtained from students in a "High 
Potential" program comprising students who do not meet regular college 
entrance requirements but who are suspected to have higher ability than 
indicated by their achievement records. 

In later analyses, the data used will be drawn from community col- 
lege students who have been admitted unprovisionally. The original goal 
of the AID project was to prepare a curriculum for community college 
students, and these new data will provide a better assessment of the 
curriculum. 

Second, an unexpected result is the proportion correct for multiple- 
choice exercises as compared with constructed responses. The proportion 
correct is markedly lower, 54.9$, for multiple -choice exercises compared 
with 66.6$ for constructed responses. In presenting a lesson-by- lesson 
comparison of the proportion correct for multiple-choice and constructed 
responses Table 5 shows this result to be quite stable across the lessons. 
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Table 4 



Number and Percent of Correct First Responses in Lessons 1 to 21 
(Excluding Lessons 7, 14, 21) 





Total correct 
first responses 


Total 

first responses 


Percent 

correct 


Multiple - Choice 
1 Correct Choice 


436 


718 


60.7 


Multiple- Choice 
2 Correct Choices 


60 


160 


37.5 


Mult iple- Choice 
3 or More Correct 
Choices 


59 


120 


49.2 


Multiple-Choice 
Correct Choice: NONE 


23 


55 


41.8 


Total Multiple -Choice 
Exe rcises 


578 


1053 


54.9 


Yes-No Exercises 
(except opinion 
questions) 


99 


133 


74.4 


True -False 


215 


254 


84.6 


Other Implied- Choice 


47 


59 


79» 7 


Total Implied- Choice 


361 


446 


80.9 


Predicted AID Response 


1282 


1861 


68.9 


Constructed AID Command 


497 


798 


62.3 


Reported Result of AID 
Use 


557 


878 


63.4 


Other Constructed- 
Response Exercises 


1003 


1476 


68,0 


Total Constructed 
Response 


3339 


5013 


66.6 



Totals 



4278 



6512 



65.7 



Table 5. 



Percent of Correct First Responses for All Multiple -choice and All 
Constructed-response Exercises in Lessons 1 to 21 
(Excluding Lessons 21) 



Lesson 


Percent correct 
multiple -choice 


Total responses 
multiple- choice 


Percent correct 
constructed response 


Total responses 
constructed response 


i 


66.7 


81 


68.0 


50 


2 


67-3 


153 


58.2 


ll*6 


3 


32.7 


101 


65.1 


373 


1* 


57- ^ 


61 


71-5 


368 


5 


1 * 6.7 


75 


70.2 


601 


6 


1*1*. 7 


76 


69.5 


298 


8 


30.8 


13 


67.2 


862 


9 


26.3 


19 


70.1 


1*72 


10 


76.3 


76 


81.8 


357 


. 11 


70.6 


17 


73.6 


208 


12 




0 


77-8 


99 


13 


5b. 3' 


35 


58.1 


296 


15 


61.2 


85 


55-3 


237 


16 


1+7-6 


81* 


65.2 


17 S 


17 


66.7 


12 


0.0 


55 


18 


1*8.1 


81 


50.6 


77 


19 


25.0 


12 


67 . 1 * 


135 


20 


61.1 


72 


51-7 


201 



5M 



Totals 



1053 



66.6 



5013 



Third, the proportion correct for implied- choice exercises (80 .9$) 
is higher than either multiple- choice or constructed- response exercises. 
This result might be expected since all of the implied-choice exercises 
imply only two choices, and about 50$ of the answers would be correct if 
they were selected by random guessing. 

Fourth, the order of percent correct among the different kinds of 
multiple- choice exercises is not what would be predicted from the number 
of correct choices available. Exercises in which there are more than 
one correct choice are more difficult (119/280 = 42.5$) than are exercises 
in which exactly one choice is correct (459/773 = 59*^$) • However, 
exercises in which none of the listed choices is correct are more dif- 
ficult (4l.8$) than the other single choice exercises (60.7$), and 
exercises in which there are three or more correct choices are easier 
(49.1$) than those in which there are two correct choices (37*5 $) • 

A more detailed analysis of the data for multiple -choice exercises 
will be discussed later in this report. 

Student Control of Amount of Instruction 

Two features of the course that allow the student to control the 
amount of instruction he receives are "hints" and "tells." Most of the 
exercises have one or more optional hints which can be requested by the 
student at any time, either before or after making a response. Also, in 
all exercises for which there is a correct answer the student may request 
the correct answer, i.e., a tell, at any time. 

How these two controls are used is as yet undetermined, and will be 
the subject of future research. The item analyses of first responses, 
on which this report is based, does provide some evidence, however. The 



number of hints and the number of tells requested before first responses 
were tallied. 

In some exercises , the hints provide information vital to the solu- 
tion of the problem. This information was put into hint messages rather 
than into the problem text itself whenever it could be assumed that a 
fairly large proportion of the students would already know it. For 
example, if the exercise is to use AID in calculating the volume of a 
cylinder of given radius and height, it is essential to use the formula 
that gives volume in terms of radium and height. Many students can be 
assumed to know this formula so it was not included in the problem state- 
ment but instead was included as the first hint. In other cases, the 
hints suggest a strategy to be' used in solving the problem. It was 
assumed that hints would be used freely by as many as a quarter of the 
students. 

Table 6 shows the number of hints requested before the first response 
by problem type. Only 89 hints preceding first responses were requested 
for Lessons 1 to 21. Since there were over 6,500 first responses for 
these lessons, this number of requests for hints is surprisingly low. 
Variation between different lessons and different problem types should 
be viewed with skepticism because of the small number of requests. 

It may be that students do not request hints until they have made 
at least one try, or they may believe they would be penalized for re- 
questing hints, or they, may not understand how to ask for them. Because 
the low number of hint requests is so unexpected it is worth pursuing 
in later research on use of control features. 
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Table 6 



Number and Percent of Hints Requested in Lessons 1 to 21 
(Excluding Lessons 7 > 14, 21) 





Total hints 
requested 


Total 

responses 


Percent 

hints 


Multiple- Choice 
1 Correct Choice 


4 


718 


0.6 


Multiple- Choice 
2 Correct Choices 


0 


160 


0.0 


Multiple-Choice 
3 or More Correct 
Choices 


0 


120 


0.0 


Multiple -Choice 
Correct Choice : NONE 


0 


55 


0.0 


Total Multiple -Choice 
Exercises 


4 


1053 


0.4 


Yes -No Exercises 
(except opinion 
questions) 


7 


133 


5.3 


True -False 


0 


254 


0.0 


Other Implied- Choice 


1 


59 


1-7 


Total Implied*-Choice 


8 


445 


O - 


Predicted AID Response 


30 


1861 


1.6 


Constructed AID Command 


19 


798 


2.4 


Reported Result of AID 
Use 


17 


878 


1.9 


Other Constructed- 
Response Exercises 


11 


1476 


0.7 


Total Constructed 
Response 


77 


5013 


1.5 


Totals 


89 


6512 


1.4 



Requests for the correct answer were expected to be relatively low 
as a first response, about 5$ as a first response and about 10$ as a 
second, or later, response. Table 7 shows the number of first responses 
requesting the correct answer by problem type. Two hundred forty- two of 
the 6,512 first responses, or 3*7 $> were requests for the answer. It is 
interesting that there was a greater proportion of such requests for 
multiple-choice exercises (4.6$) than for constructed- response exercises 
(3-8$), indicating that multiple -choice exercises are more difficult. 

In the analysis of structural variables affecting problem difficulty, 
proportion correct is taken as the measure of difficulty. An interesting 
comparison could be made by using the number of requests for answers as 
a measure of difficulty. This was not done in the current analysis be- 
cause the number of subjects was too small to permit meaningful application 
of the stepwise multiple linear regressions used below. 

Lesson to Subtest Relationship 

The percentages correct by lesson listed in Table 8 show little 
variation, with the exception of Lesson 17, and it can be inferred that 
the curriculum is reasonably consistent in difficulty. Lessons 10 and 12 
appear to be the easiest, and Lessons 17 and 18 seem to be the most 
difficult. 

Two of the more difficult lessons are tests (Lessons 13 and 20), 
and the average percent correct for the three tests (Lessons 6, 13, and 
20) is 60.7 $, which is slightly lower than the average for tutorial 
lessons, 63*3 $• To compare tutorial lessons and tests, each test was 
divided into five subtests each consisting of the exercises associated 
with one of the five preceding tutorial lessons, and the percent correct 
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Table 7 



Number and Percent of Requests for the Correct Answer (Tells) 
in Lessons 1 to 21 (Excluding Lessons 7 > 14, 21 ) 





Total tells 
requested 


Total 

responses 


Percent 

tells 


Multiple- Choice 
1 Correct Choice 


39 


718 


5.4 


Multiple - Choice 
2 Correct Choices 


5 


160 


3.1 


Multiple - Choice 
3 or More Correct 
Choices 


3 


120 


2.5 


Multiple - Choice 
Correct Choice : NONE 


2 


55 


3-6 


Total Multiple- Choice 
Exercises 


49 


1053 


4-7 


Yes-No Exercises 
(except opinion 
questions) 


1 


133 


0.8 


True -False 


0 


254 


0.0 


Other Implied- Choice 


0 


59 


0.0 


Total Implied- Choice 


1 


446 


0.2 


Predicted AID Response 


82 


1861 


4.4 


Constructed AID Command 


34 


798 


4.3 


Reported Result of AID 
Use 


14 


878 


1.6 


Other Constructed- 
Response Exercises 


62 


1476 


4.2 


Total Constructed 
Response 


192 


5013 


3-8 


Totals 


242 


6512 


3-7 



Table 8 



Comparison of Scores in Lessons with Scores in 
Related Exercises in Subtests 



Lesson 

number 


Proportion correct 
in lesson ($) 


Proportion correct 
in test ( °jo ) 


1 


67.9 


85 .O 


2 


63*3 


75.0 


3 


58.2 


71.4 


4 


69.4 


46.7 


5 


67.6 


52.7 


8 


66.7 


56.0 


9 


68.4 


56,2 


10 


80.8 


54.4 


11 


73.3 


80.6 


12 


77.8 


52.4 


15 


66.6 


61.7 


16 


61.0 


59.1 


17 


11.9 


51.5 


18 


49.4 


52.7 


19 


67.4 


54.6 


Mean 

S.D. 

N 


63.3 

16.1 

15 


60.7 

11.7 
15 



Correlation coefficient (R) = . 165 

R 2 = .027 
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on each subtest was compared with the proportion correct on the lesson 

tested by that subtest. Percentages correct on these subtests are given 

in the second column in Table 8 and are associated with the appropriate 

lesson. There is essentially no correlation between percent correct on 

o 

lessons and percent correct on the associated internal tests (R = .027). 
The predictive power of lesson scores for individual students will be 
explored in a later analysis of individual performance. 

Multiple -choice Exercises 

In the first 21 lessons of the AID course there are 80 multiple- 
choice exercises. In all of these exercises the choices are listed and 
labeled with letters, and if there is more than one correct choice, the 
students are expected to type the labels for all of. the correct choices. 
Two formats are used: •• 



(Vertical) 



A. TYPE 

B. DELETE 

C. SET 

D. DO 

N. NONE OF THE ABOVE 



(Horizontal) 



A 



B 



1.2 SET X=1 

1.3 TYPE Xt2 

1.4 TYPE Xt3 



1.2 DEMAND X 

1. 3 TYPE Xt2 

1.4 TYPE Xt3 



Generally, the vertical form is used if each choice can be printed 
on one line, and the vertical form is used if several lines are required 
to print single choices. 



Ic3 
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The data for multiple -choice exercises are divided into four classes, 
according to the number of correct choices (one, two, more than two, none). 
The number of correct first responses, total number of first responses, 
and percent correct on first responses for each problem type are shown 
in Table 4 presented earlier. The class of multiple -choice exercises 
with one correct choice is sufficiently numerous (5 6 exercises) to warrant 
a more detailed inspection of the data. These exercises were subdivided, 
into the 10 following classes, and proportion correct was calculated for 
each of these classes. 

(a) Algebraic Equivalence I. The student is given an algebraic 
expression and is asked to choose an equivalent expression. 
Example : 

TYPE 10/7 - 5 - 2 
COULD BE WRITTEN 

A. TYPE 10/(T-5) - 2 

B. TYPE (10/7) - (5-2) 

C. TYPE (10/7) - 5 - 2 

N. NONE 

(b) Algebraic Equivalence II. The student must choose a described 
algebraic expression. 

Example : 

WHICH COMMAND WILL CAUSE AID TO MULTIPLY 25 

BY 5 AND DIVIDE BY 3? 

A. TYPE 25 X 5/3 

B. TYPE 25 *• 5/3 

C. TYPE 25(5/3) 

N. NONE OF THE ABOVE 
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(c) Choice of AID Programs. The student must choose which of two 
routines will produce a specified result. 

Example : 

WHICH PART TYPES THE SMALLEST OF TWO NUMBERS? 

A. 3.1 TYPE R IF R > S 

3.2 TYPE S IF R <= S 

B. 4.1 TYPE R IF R < S 

4.2 TYPE S IF R >= S 

(d) Mechanics. The student is asked about the mechanics of using 
the instructional program or the AID interpreter. 

Example : 

WHICH COMMAND WILL STOP THE AID INTERPRETER, AND 
RETURN YOU TO YOUR LESSON? 



A. 


CTRL-H 


B. 


RETURN 


C. 


INST 


D. 


CTRL-T 



(e) Syntax of AID Commands. The student must decide which of a 
list of commands is syntactically correct. 

Example : 

WHICH OF THESE COMMANDS WILL CAUSE AID TO STOP 



AND 


wait FOR YOU TO 1 


A. 


3.7 


ASK S = *• 


B. 


3.7 


DEMAND S 


C. 


3.7 


REQUEST S = 


D, 


3.7 


DEMAND S 



(f) Semantics of AID Commands. The student must choose from among 
syntactically correct commands those that effect a specified 
action. 

Example : 

WHICH COMMAND(S) WILL NEVER GIVE A NEGATIVE 
NUMBER NO MATTER WHAT THE VALUE OF X IS? 

A. TYPE X/5 

B. TYPE SGN(X)*(X/5) 

C. TYPE X/(-5) 

D. TYPE SGN(X)*(X/(-5)) 

N. NONE 

(g) Boolean Equivalence I. A Boolean statement is given, and the 
student must choose an equivalent statement. 

Example : 

X <= M 

MEANS THE SAME AS WHICH OF THESE? 



A. 


M # X 


B. 


M <= X 


C. 


M >= X 


N. 


NONE 



(h) Boolean Equivalence II. The student must choose a Boolean 
statement, given a description of it in English. 

Example : 



WHICH 


MEANS "Q IS NON-NEGATIVE? 


A. 


Q 


o 

A 


B. 


Q 


>= 0 


C. 


Q 


A 

o 


D. 


Q 


<= 0 


N. 


NONE 
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(i) Sequence of Execution. The choices are lists of step numbers , 
and the student must choose the list that executes the commands 
in a specified order. 

Example : 

16.1 DO STEP 2.1 IF Q < 0 

16.2 TYPE Q 

2.1 SET Q = - Q 
DO PART 16 

IF Q = 3> THE STEPS WILL BE DONE IN WHICH ORDER? 



A. 


B. 


C. 


D. 


16.1 


16.1 


l6.l 


16.1 


2.1 


16.2 


2.1 


16.2 


16.2 


2.1 







(j) Miscellaneous 

The number of exercises , number of first responses , and percent 
correct on first responses for each of the above classes of multiple- 
choice exercises are given in Table 9* 

It is difficult to draw firm conclusions from this subdivision of 
problems but these data present some clues to student behavior. The two 
classes with the highest proportion correct are (d) Mechanics, and (g) 
Boolean Equivalence I. The two classes with the lowest proportion cor- 
rect are (c) Choice of AID Programs and (h) Boolean Equivalence II. The 
proportion correct for class (i) Sequence of Execution is only 51*6$ over 
13 exercises. It may be that the students have difficulty understanding 
order of execution of commands and that the curriculum should emphasize 
this area. The data from later lessons on loops will be studied to see 
if they also suggest some revision in this aspect of the course. 
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In .multiple- choice exercises, several variables other than problem 
type may contribute to difficulty. One of these variables is the number 
of choices given; presumably the larger the set of possible answers, the 
more difficult the choice. The range of choices in these exercises is 
from two to five. To estimate the effect of the number of choices, the 
correlation between proportion correct and number of choices was calcu- 
lated, using only those exercises (49 of the 56 ) for which there were 10 
or more first responses. This correlation was .000, indicating that the 
number of choices bears no linear relationship to problem difficulty. 
This result is not conclusive since number of choices and whether an 
exercise is in Class (c) or (d) are statistically dependent. Further, 
Class (c) is the most difficult class but has an average of only two 
choices per exercises, whereas Class (d) is the least difficult class 
but has an average of 4.3 choices per exercises. 

A further analysis was made by using multiple step-wise linear 
regression with observed proportion correct, x., as the dependent vari- 
able, and using the following independent variables: 
number of choices 



x. 



x. 



x, 



1 if one of the choices is 

M. NONE OF THE ABOVE 

0 otherwise 

t 

1 if the exercise is in Class (a) Algebraic Equivalence I 

0 if not 

1 if the exercise is in Class (b) Algebraic Equivalence II 

0 if not 

1 if the exercise is in Class (c) Choice of AID Programs 
0 if not 
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x^ 1 if the exercise is in Class (d) Mechanics 
0 if not 

Xg 1 if the exercise is in Class (e) Syntax of AID Commands 
0 if not 

x 1 if the exercise is in Class (f) Semantics of AID Commands 
9 

0 if not 

x^ 1 if the exercise is in Class (g) Boolean Equivalence I 
0 if not 

X- |1 1 if the exercise is in Class (h) Boolean Equivalence II 

0 if not 

x^ 1 if the exercise is in Class (i) Sequence of Execution 
0 if not 

x.. 1 if the exercise is in Class (j) Miscellaneous 

0 if not. 

This regression was computed for the 49 exercises for which there were 

10 or more first responses. In Table 10, the variables are listed in 

the order in which they were "stepped" into the regression equation, and 

2 

the values of R and R are given. The variables in the regression ac- 
counted for 38 $ of the variance of the observed proportion correct. This 
leaves 62 $ of the variance unaccounted for, indicating that there are 
other variables, as yet unidentified, that affect problem difficulty in 
this group of exercises. 

The two variables that entered into the regression first, x^ and x^, 
are the variables for membership in Classes (d) and (c), the same two 
classes that are statistically dependent on number of choices, and that 
the thiyd variables to enter into the regression is Xg, the number of 
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Table 10 



Order in Which Independent Variables Were Entered into a 
Multiple Stepwise Regression with the Associated 
Correlation Coefficients (R) 



Order 


Variable 


R 




Increase 


1 


*7 


0.33 


0.11 


0.11 


2 


x 6 


o.4o 


0,16 


0.05 


3 


X 2 


0.52 


0.27 


0.11 


4 


x 5 


0.54 


0.29 


0.02 


5 


x 8 


0-55 


0.31 


0.02 


6 


X 3 


0.56 


0.32 


0.02 


7 


x io 


o.59 


0.35 


0.03 


8 


X 12 


0.60 


0.36 


0.01 


9 


x 13 


0.61 


0.37 


0.01 


10 


X 11 


0.61 


0.38 


0.01 


11 


x 4 


0,61 


0.38 


0.00 


12 


x 9 


0.61 


0.38 


0.00 



choices, x^ and Xg together accounted for 16$ of the variance in the 
observed proportion correct, whereas the addition of increased that 
figure by 11$, indicating that the number of choices is of significant 
effect even after Classes (c) and (d) are taken into account. The fact 
that x 2 entered into the regression before any of the other variables 
(except x^ and x^) supports the hypothesis that the number of choices 
contributes to problem difficulty. 

Constructed- response Exercises 

The constructed-response exercises fonn by far the most numerous 
class of exercises in the course (390 out of 675), and represent more 
diversity than either multiple- choice or implied- choice exercises. One 
reasonably homogeneous subclass of the constructed-response exercises, 
the "predicted AID response" exercises, was chosen for more detailed 
study. This class contains 135 exercises, and the data contains l,86l 
responses to be analyzed. 

In the "predicted AID response", exercises, the student is asked to 
predict the result if a given command were to be executed by the AID 
interpreter. For these exercises, a simple classification scheme is not 
possible because of the interaction of variables that contribute to the 
relative difficulty of each exercise. For this reason, multiple step- 
wise linear regression was used to find which of these variables had the 
greatest influence on problem difficulty. Only the 104 exercises with 
10 or more responses were used in this study. The dependent variable, 
x^, to be pj’edicted by the regression equation was observed proportion 
correct. Eleven independent variables were identified and values assigned 
to each exercise. The independent variables were the following: 

qx 
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Xg Number of arguments for TYPE commands. The problem may 

contain several TYPE commands, each of which can have one 
or more arguments. The number of arguments for all TYPE 
commands are summed to give the value of x^. 
x^ Number of AID commands. All of the commands displayed in 
the problem are counted for x^. If the exercise is a con- 
tinuation of a preceding exercise or exercises, the commands 
displayed in the preceding exercise may also be counted 
into this variable. Variables x^ to x^ are concerned 
with this extended set of commands, 
x^ Proportion of AID commands to be used. Some commands may- 
be extraneous, x^ is obtained by dividing the number of 
commands needed by the total number of commands displayed. 

x c Number of function calls. x_ is the total number of func- 

5 ? . 

tion calls in the displayed commands, including both defined 
functions and standard AID functions, 
x^ Number of clauses. The number of. IF, FOR, and TIMES clauses 
are counted. 

x^ Number of substitutions required, x^ is a count of the 

substitutions required for the correct solutions, including 

substitution of numbers for real variables and substitution 

of expressions for real variables, substitution of function 

definitions is not counted. As an example, suppose these 

commands were given: 

LET F(X) = X+2 
SET A = 5 
TYPE F( 27/A) 

68 
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The substitution of 5 for A is counted, and the substitu- 
tion of 27/A for X in the expression X+2 is counted, so 
the value of x^ is 2. 

Xg Imbeddedness of arithmetic expression. The student may 

be asked to evaluate some arithmetic expression, possibly 
after performing one or more substitutions. To measure 
imbeddedness, we used a "completed" expression, which is 
obtained from the expression displayed by making all re- 
quired substitutions and by inserting all implicit paren- 
theses. This completed expression is the basis for 
calculating variables Xg and x^. For Xq, imbeddedness is 
measured by finding the maximum number of left parentheses 
that can be counted before a right parenthesis is encountered. 
As an example, consider these commands: 

SET X = 2 
TYPE X + Xt3 

The completed expression derived from this would be 
2 + (2t3) 

and the imbeddedness variable Xg would have the value 1 
since there is only one pair of parentheses, 
x Number of operations. The value of x is found by counting 

y y 

the number of operation symbols in the completed arithmetic 
expression. 

x^q Number of exponentiation operators. The occurrences of 
the symbol t in the extended set of displayed commands 
are counted for x^. 
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x^ Number of implied parentheses. Here the completed arith- 
metic expression is not used for a basis for calculation. 
What is counted is the number of pairs of parentheses that 
are used implicitly. For example, in this command, 

SET X = 2f3/b , 

there is one pair of implied parentheses, (2t3)/4, and 
the value of x^ is one . 

x 1p Number of preceding "predicted AID response" exercises in 
the same lesson. 

Table 11 shows the order in which the variables were "stepped" into 

2 

the regression equation and the values for R and R . The variables used 
accounted for only 21$ of the variance in proportion correct. The vari- 
able with the greatest effect was x , the number of operations. This 

9 

variable is related to, but not identical with, the operations variable 
used by Loftus (1970) and by Jerman ( 1971) in similar models of problem- 
solving difficulty. In both cases, the operations variable was found to 
be significantly related to difficulty. The second most effective 
variable was x^, the number of clauses, and the third was x^, the number 
of function calls. 

The variables x ^ , x^, x^, x^ and Xg all added essentially nothing 
to the value of R . In fact, the first six variables accounted for 20$ 
of the variance and the next four together added only 1 % more. 

8. Analysis of Selected Aid Data 

Although this report is concerned primarily with data collected by 
INST, a large body of data was also collected by the AID interpreter, 
and a sample of this data was selected and analyzed by hand. These data 



Table 11 



Order in Which Independent Variables Were Entered into a 
Multiple Stepwise Regression with the Associated 
Correlation Coefficients (R) 



Variable 


R 


E 2 


2 

Increase in R 


x 9 


0.30 


0.09 


0.09 


x 6 


0.41 


0.17 


0.08 


X 5 


0.43 


0.18 


0.01 


O 

i — 1 


0.43 


O.19 


0.01 


X 12 


0.44 


0.19 


0.00 


x 4 


0.44 


0.20 


0.01 


*8 


0.45 


0.20 


0.00 


X 3 


0.45 


0.20 


0.00 


*7 


0.45 


0.21 


0.00 


x 2 


0.46 


0.21 


0.00 


i — l 
i — l 


0.46 


0.21 


0.00 
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were collected by the AID interpreter as students used it to solve prob- 
lems given by the instructional program. The data consists simply of the 
characters typed by the student together with necessary bookkeeping 
information such as student number, problem identifiers, and date. 

The problem of analyzing these data from the AID interpreter is 
basically unsolvable. As was discussed earlier, no effective procedure 
exists that can determine if a student -written program is equivalent to 
a given, correct program. An approximation of this procedure may be 
possible, but for the moment data collected by the AID interpreter must 
be analyzed by hand. However, no evaluation of the course will be com- 
plete without an examination of the programs written by the students. 

In a later report, more sophisticated and extensive analyses of the AID 
interpreter data will be made. 

In the first 21 lessons of the course, there are 58 exercises that 
require the student to use the AID interpreter. These exercises vary 
greatly in difficulty. Some simply ask the student to call the AID in- 
terpreter and copy a single command; others ask the student to apply 
recently learned principles to solve complex problems. Of the 58 exer- 
cises requiring the use of AID, 28 are essentially copying tasks. Of 
the remaining 30 exercises, there are approximately 20 that can serve as 
genuine tests of the students* ability to use the AID interpreter. Eight 
of these exercises were chosen for analysis, the results of which are 
summarized in Table 12. 

All these exercises follow the same form. The instructional pro- 
gram presents a problem and asks the student to use the AID interpreter 
to solve it (the student may request hints at this point, but he cannot 



give an answer, until he has called the AID interpreter). Next, the 
student calls the AID interpreter and attempts to solve the problem. 

This step is called the "first use of AID" in Table 12. The student may 
at this time write a program, try it out, replace one or more commands, 
correct typographical errors, etc. Finally, when he is satisfied with 
his solution, the student switches back to the instructional program and 
is asked for a report on the results he obtained. If his reported re- 
sults are incorrect, the instructional program gives him additional 
instruction and asks him to switch back to AID and tiy again. 

Table 12 shows the number of students who correctly solved each 
problem on the first try and the number of students who eventually ar- 
rived at a correct solution. The percent correct for first AID use is 
quite low (29*9$) • Since these problems are among the most difficult in 
the entire course, this percent is expected to be lower than the 66$ 
average for other exercises. Because of the additional help given by 
the instructional program after the first use of AID, the percent correct 
is expected to rise when subsequent uses are considered, and, in fact, it 
does rise to 68 . 4 $. ' The difference between proportion correct for first 
try and for subsequent tries is most pronounced for Lesson 5 > Exergise 30. 
Only two out of 13 students solved the problem on the first try, but 12 
of them solved it eventually. 

During the hand-grading of these exercises, notes were made on the 
most common errors for each problem. Following are the problem state- 
ments (including subproblems) for each of the eight exercises, including 
comments on the errors observed while the exercises were being graded. 
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Problem 5-30: 



1 CENTIMETER = .3937 INCHES. START AID AND CONVERT 
THE FOLLOWING LENGTHS TO CENTIMETERS: 

6.9 INCHES 
7.445 INCHES 
23.9753 INCHES 

Problem 5-30.1: 

FROM THE ABOVE CALCULATIONS, 

6.9 INCHES = ??? CENTIMETERS 

Problem 5-30.2: 

7.445 INCHES = ??? CENTIMETERS 
Problem 5-30.3: 

23.9753 INCHES = ??? CENTIMETERS 

The students must solve this problem with direct TYPE and SET 
commands since indirect (stored) commands have not yet been introduced. 

One solution would be 

SET X = .3937 

TYPE 6 . 9 /X, 7.445/X, 23.9753/X 

On this problem only two of 13 students were correct on the first 
try. Seven of the students made algebraic errors; six used multiplication 
instead of division, and one used .3937/6.9 instead of 6 . 9/* 3937* Tw° 
students made algebraic errors on the second try, one repeated the error 
of using multiplication rather than division, and one came up with a 
unique use for x (6.9/3* l4l6) . 

Although a number of syntax errors were made, they were almost all 

i 

typographical errors and were corrected by the student without any 
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intervening instruction. Only one student was unable to correct his 
errors in syntax. It is likely that had the students known the correct 
algebraic formulation, all but one would have produced a correct solution 
on the first try. 

Problem 5-31: 

TO FIND THE NEW AMOUNT IN A SAVINGS ACCOUNT, CALCULATE THE 
INTEREST AND ADD IT TO THE LAST BALANCE. START AID AND 
CALCULATE THE INTEREST AND THE NEW BALANCE AFTER ONE YEAR 
FOR AN ACCOUNT WITH AN INTEREST RATE OF 4.5 PERCENT PER 
YEAR AND A PREVIOUS BALANCE OF $3274.86. (ASK FOR A HINT 
IF YOU NEED ONE.) 

Problem 5-31*1: 

WHAT IS THE INTEREST ON THE ABOVE ACCOUNT TO THE NEAREST 
PENNY? 

Problem 5-31-2: 

WHAT IS THE NEW BALANCE IN THE ACCOUNT? 

This problem can be done in several ways. One solution, which was 

not used by any of the students, would be 

SET P = 3274.86 
SET I = .045 *-P 
TYPE I, P+I 

Only three out of 12 students were correct on their first try, but 
11 eventually succeeded. Two students made syntax errors of the same 
kind; they both used the symbols $ and $ in an arithmetic expression. 

Two students calculated the new balance directly, without calculating 
the interest independently, which left them unable to answer the question 
in the first subproblem. Obviously, the problem should have been explicit 



in its request for two independent calculations. Four students used 4.5 
as the interest rate rather than .045, and two students used the wrong 
arithmetic operation. Of the 11 students who eventually gave correct 
answers, only four used variables. Here again it would seem that the 
major difficulty is with algebra, rather than with the use of AID. 

Problem 8-9: 

START AID AND USE A "LET" COMMAND TO DEFINE A FUNCTION 
THAT GIVES THE RECIPROCAL OF X. USE YOUR FUNCTION TO 
FIND THE RECIPROCAL OF 

119-4 

67.3*3 

6+4 

Problem 8-9« l! 

WHAT IS THE RECIPROCAL OF 119 . 4 ? 

A correct solution of this exercise would be 
LET F(X) » VX 

TYPE f(119.4), f( 67 . 3 * 3 ), f(6+4) 

/ 

Over half of the students solved this on the first try, and three- 

quarters of them were eventually correct. The LET command was introduced 

in Lesson 1 8, and this was the second exercise in which the students used 

AID in this lesson. Five out of 20 students made syntax lerrors that 

they could not correct, all in the LET command. Three students made 

2 

algebraic errors, using as the formula for reciprocal .1 x, x /2, and 
2 

1/x . Two students who defined the function correctly redefined it 
needlessly before each function call. 
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In this exercise , the syntax of the newly introduced LET command was 
the source of most errors. 

Problem 8-27 • 

USE AID TO DO THIS PROBLEM. DEFINE A "VOLUME" FUNCTION 
THAT WILL GIVE THE VOLUME OF A CYLINDRICAL TANK OF RADIUS 
R AND HEIGHT H. 

(VOLUME = 3,l4l6 TIMES THE RADIUS SQUARED TIMES THE HEIGHT. J 

FIND THE VOLUMES OF 2 TANKS: 

TANK A IS 57 . 5 FEET HIGH AND HAS A RADIUS OF 
18.6 FEET. 

TANK B IS 65.4 FEET HIGH AND THE RADIUS IS 

19.3 FEET. 

Problem 8-27*1: 

WHAT IS THE VOLUME OF TANK B? 

Problem 8-27-2: 

WHICH TANK HOLDS MORE, TANK A OR TANK B? 

A correct solution for this exercise is 

LET V(R,H) • 3.1416 * Rt2 -* H 
typev(i8 .6, 57.5) , V(19.3, 65.4) 

Only four of 16 students succeeded on this exercise on the first 
try, and only six eventually succeeded. Seven students did not define a 
function at all but solved the problem arithmetically. Most of the stu- 
dents who tried to use a function were confused about the syntax of the 
LET command, the use of functions of more than one variable, or the use 

of dummy variables. Two students used the wrong algebraic formula (jtrh 
2 

instead of nr h) even though the problem gave the formula. 



Here it seems the course moved too fast in introducing functions of 
more than one variable so soon after simple functions were introduced. 

Some of the preceding exercises which required substitution of expressions 
containing real variables for the dummy variables used in function defin- 
itions may have exacerbated an existing confusion about variables. The 
next exercise which required a function of only one variable proved much 
less difficult. 

Problem 8-28; 

USE AID TO DO THIS PROBLEM. DEFINE A FUNCTION TO CONVERT 
DEGREES FAHRENHEIT TO DEGREES CENTIGRADE. THEN CONVERT 
THESE TEMPERATURES TO CENTIGRADE: 

0, 10, 32, 72, 212 

Problem 8-28.1: 

72 DEGREES FAHRENHEIT IS EQUAL TO HOW MANY DEGREES 
CENTIGRADE? 

A correct solution is 

LET C(F) = (F-32) ->4 5/9 

TYPE C(0), C(10), C(32), C(72), C(212) 

and one -third of the students gave this solution, or an equivalent one, 

on their first try. All of the students arrived at a correct solution 

on a later try. 

Seven out of 12 students made algebraic errors, five of them iden- 
tical: x - 32 ->4 5/9 instead of (x-32) -*• 5/9* Omission of necessary 
parentheses in an algebraic expression with only two operations indicates 
a basic confusion about the hierarchy of operations. 
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Problem 9~3- 



WRITE A FUNCTION, H(A,B), THAT WILL FIND THE HYPOTENUSE 
OF A RIGHT TRIANGLE IF THE LENGTHS OF THE OTHER TWO SIDES 
ARE GIVEN BY A AND B. • START THE AID INTERPRETER AND TEST 
YOUR FUNCTION ON THESE TRIANGLES; 

1. A=3, B=4 

2. A=12, B=12 

3* A=l/2, B=3A 

4. A=9, B= 13-2 

Problem 9 - 3 . 1: 

WHAT IS THE HYPOTENUSE OF THE FIRST TRIANGLE ABOVE? 

(WHERE A=3, B=4) 



Problem 9-3*2: 

WHAT IS THE HYPOTENUSE OF THE TRIANGLE WITH SIDES 
A=I/2 AND B=3/4? 



This is the first exercise using AID after the standard AID function 

SQRT is introduced, and the correct solution is 

LET H(A,B) = SQRT(At2 + Bt2) 

TYPE H(3,4) , H( 12,12), H(1/2,3A), H(9,13*2) 

The students did very poorly on this problem. Three of the 18 were 
correct on their first tiy, but only seven eventually achieved a correct 
solution. Most of the errors were syntax errors either in the LET com- 
mand or in the function call. Like Problem 8-27 , there were a number of 
errors in the use of parentheses: 

1/2 t2 + 3/4 t2 was used for (l/2)t2 + (3/4)t2 
SQRT( At2) + ( Bt2) was used for SQRT( At2 + Bt2) 

SQRT A was used for SQRT(A) 
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SQRT(A)t2 + (B)t2 was used for SQRT(At2 + Bt2) 
9l6.25tI/2 was used for 9l6.25t(l/2) 

Several students did not know the correct formula and used: 



Three students did not define a function but did produce correct 
specific solutions for all four triangles. 

Problem 12-4: 



START AID AND WRITE A PROGRAM (PART) THAT WILL ASK YOU 
FOR 3 NUMBERS A, B, AND C, AND THEN GIVE YOU THE AVERAGE 
OF THE 3 NUMBERS. AFTER YOU HAVE TESTED YOUR PROGRAM, 
USE IT TO FIND THE AVERAGE OF 

A = 179.053 
B = 23.7 

c = 271.0015 



Problem 12-4.1: 

WHAT ANSWER DID YOU GET? 

This is the first exercise in the eight hand-graded exercises that 
requires the use of a stored program. One correct solution is 

4.1 DEMAND A 

4.2 DEMAND B 

4.3 DEMAND C 

4.4 TYPE (A+B+C)/3 

This program would be executed by giving the command 

DO PART 4 

Six of 15 students solved this problem on the first try, and another 
three succeeded later. The majority of errors were algebraic; seven stu- 
dents used incorrect formulas: 



(Va + t/b) 2 
(A 2 ):+ (B 2 ) 
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A + B + 1/3 
ABC/3 

3/A + B + C 



(used by four students) 
(used by two students) 
(one student) 



Of the nine correct solutions, four were not debugged, and the 
others were tested for only one set of poorly chosen values. Here again, 
as in Problem 9-3; some students produced specific but not general 
solutions. 

Problem 15-15: 



Problem 15-15-1: 

DID YOUR PROGRAM WORK? 

An economical solution to this problem is 

9.1 DEMAND X 

9.2 DEMAND Y 

9. 3 TYPE X IF X < Y 

9-4 TYPE Y IF Y <= X 

For this exercise, two out of 11 students were correct on their 
first try, and only three did not solve the problem at all. 

The greatest difficulty was in. inputting values for the two vari- 
ables. Eight students had trouble with this. Several of them tried 
rather ingenious (but incorrect) variants of the FOR clause: 



Four of the eight correct solutions were well debugged, three were 



USING AID, WRITE A PROGRAM THAT WILL FIND THE SMALLER 
OF TWO NUMBERS X AND Y. TRY SEVERAL DIFFERENT VALUES 
OF X AND Y. 



DO PART 9 FOR X = 6, Y = -2. 

DO part 9 for (x,y) = (4,7), (8,4), (4321,6493). 



tested for only one set of values, and one was not tested at all. 



The above discussion of the performance of students on eight se- 
lected exercises is not intended as a definitive survey or as a final 
evaluation of the curriculum but as an indication of the ways students 
solve problems and of guidelines for future analyses of student inter- 
action with the AID interpreter. It is clear that some analysis of errors 
is desirable and leads to more meaningful interpretation than correct- 
incorrect grading alone. 

In summary, the most common difficulties of the students involved 
algebra more than programming and centered around the following aspects: 

(1) Use of parentheses. Whether or not students would have cor- 
rectly formulated the algebraic expressions if they were using 
ordinary algebraic notation rather than AID notation is not 
known, but one suspects that in many cases they could not have 
done so. 

(2) Use of standard algebraic formulas. The number of errors made 
in standard formulas is a strong indication that the course was 
delinquent in not providing these formulas wherever needed. 

(3) Use of functions of more than one variable, 

9 . Conclusions 

In this preliminary report, we discussed the interaction between 
students and a tutorial computer-assisted course in programming. Rates 
of progress as measured by the Daily Report program were discussed using 
one class of students as an example. A classification scheme for the 
exercises was devised, and the first responses to exercises in the first 
21 lessons were examined by problem type and by lesson. Two classes of 
exercises, one set of 49 multiple-choice exercises and one set of 104 
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constructed-response exercises, were analyzed in detail to determine the 
structural variables that affected problem difficulty. Multiple step- 
wise linear regression was used to model exercise characteristics as 
determiners of exercise difficulty. Eight exercises that required the 
students to use the AID interpreter were analyzed by hand. 

Some of the results of these investigations were unexpected, some 
confirmed predictions, and some were anomalous. First, the proportion 
of correct initial responses (65.7$) was lower than hoped for. There 
are no firm guidelines giving the optimal proportion correct for CAI. 
Some curriculum designers aim for nearly 100 $ correct, others have 
claimed 85$ to be ideal. We assumed that there is little learning if 
the success rate is very high. On the other hand, with a very low suc- 
cess rate, students are likely to become discouraged. Until further 
research suggests an optimum success rate, each curriculum designer is 
left to his own best judgment. In our case, we believed that the pro- 
portion of correct first responses should be about 75$, which is nearly 
10$ higher than the proportion correct attained by the students studied 
in this investigation. It should be noted, however, that proportion 
correct summed over students is a simplistic measure of effective in- 
struction, particularly for computer-assisted instruction where complex 
branching strategies are used, and better measures are obtained by 
examining the distributions of correct answers achieved by individual 
students over all exercises and the success with which students meet the 
goals of the course. 

Second, proportion correct among all the lessons was quite stable, 
with only four lessons in the first 21 deviating from the total 65.7$ 



correct by more than 10$. The very low 11.9$ for Lesson 17 was unexpected 
and remains unexplained. 

Third, examination of proportion correct for different problem types 
indicated that the multiple -choice exercises were more difficult than the 
constructed- response exercises (54.9 $ correct compared to 66.6$). 

Fourth, the multiple -choice exercises that were examined closely 
were those with only one correct choice, excluding the exercises for which 
the correct answer is "none of the above." The total proportion correct 
for these exercises was 60.7$, but the variation between different sub- 
classes is striking. The easiest multiple-choice exercises concerned 
the mechanics of operating the instructional program and the AID inter- 
preter; the most difficult were those in which the student had to choose 
which of two given AID programs would produce a specified result. 

Fifth., we found no simple, unambiguous relationship between dif- 
ficulty and the number of correct answers in multiple-choice exercises. 

The category labeled "multiple- choice, 1 correct choice" was easiest in 
terms of proportion of correct first responses (60.7$) > but the other 
single choice category, "multiple-choice, correct choice: none," was the 
most difficult (4l.8$). Further, the category with three correct choices 
was easier (4-9.2$) than the category with two correct choices (37»5$)* 

Sixth, a more sophisticated approach was taken in the analysis of 
performance on a selected class of constructed-response exercises. The 
exercises used were those in which the student predicts the result of 
using given AID commands. A dozen structural variables were defined and 
a stepwise linear regression performed to determine the parameters that 
could estimate the difficulty of these and similar exercises. We 




succeeded in obtaining a model that accounted for 21$ of the obtained 
variance in problem difficulty, a reasonable but not spectacular fit for 
a first model. Three structural variables (the number of operations, 
the number of clauses, and the number of function calls) accounted for 
18$ of the variance. All these variables correlated negatively with 
proportion correct. 

The investigation suggested a few general conclusions about student 
behavior. The mechanics of using the instructional system did not cause 
Undue difficulty, as shown by the performance of the students on Lesson 1 
and the subsequent test of that lesson, as well as by their performance 
on the Multiple -choice Exercise Class. It remains unclear whether or not 
the students learned to use the various control features as effectively 
as they could (the number of requests for hints was much lower than ex- 
pected), and further investigation is needed of this aspect of the. 
instructional system. 

The use of direct commands, and the concept of stored programs and 
their execution were readily mastered. The syntax of AID commands, except 
for the LET command, caused little difficulty. Although many students 
made syntactical errors in using the AID interpreter, most of these 
errors were immediately corrected by the students. 

The three areas in which the students seemed least adept were: 

(1) Hierarchy of arithmetic operations. 

(2) Use of functions, especially functions of more than one 
variable. 

(3) Sequence of execution of AID commands. 

86 
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This last observation is substantiated by the study of structural vari- 
ables in constructed- response exercises, and also by the examination of 
the student performance using the AID interpreter. Because of the dif- 
ficulty students had with the hierarchy of operations and the use of 
functions, it is reasonable to assume that they lacked necessary algebraic 
experience and that the course did not provide that experience for xhem. 

It is harder to draw conclusions from the difficulty students had with 
the sequence of execution. If this result holds true for later lessons 
and for other groups of students, it will pinpoint a major weakness in 
the course, since an understanding of the sequence of execution is 
essential in programming. We hope that in later reports more conclusive 
evidence can be supplied on these and other unanswered questions. 



« 




REFERENCES 



Davis , M. Computability and unsolvability . New York: McGraw-Hill, 1958* 

Friend, J. , & Atkinson, R. C. Computer-assisted instruction in program- 
ming: AID. Technical Report No. 164, Institute for Mathematical 
Studies in the Social Sciences, Stanford University, January 25, 1971- 

Friend, J. INSTRUCT coders' manual. Technical Report No. I 72 , Institute 
for Mathematical Studies in the Social Sciences, Stanford University, 
May 1, 1971. 

Jerman, M. Instruction in problem solving and an analysis of structural 
variables that contribute to problem-solving difficulty. Technical 
Report No. 180, Institute for Mathematical Studies in the Social 
Sciences, Stanford University, November 12, I 97 I. 

Loftus, E. J. F. An analysis of the structural variables that determine 
problem-solving difficulty on a computer-based teletype.,. Technical 
Report No. 162, Institute for Mathematical Studies in the Social 
Sciences, Stanford University, December 18, 1970- 

Mark, S. L. , & Armerding, G. W. The JOSS primer . The RAND Corporation, 
Santa Monica, California, August, 19£>T- ” 

Shaw, J. C. JOSS : Experience with an experimental computing service for 
users at remote typewriter consoles . The RAND Corporation, Santa 
Monica, California, May, 1965 . 



OFFICE OF NAVAL RESEARCH 

PERSONNEL AND TRAINING RESEARCH PROGRAMS (CODE 458) 
DISTRIBUTION LIST 



Chief of Naval Research 
Code 458 

Department of the Navy 
Washington, D. C. 20360 



Director 

ONR Branch Office 
1030 East Green Street 
Pasadena, California 91101 



Director 

Naval Research Laboratory 

Washington, D. C. 20390 

Attn: Technical Information Division 



Director 

Naval Research Laboratory 
Washington, D. C. 20390 
Attn: Library, Code 2029 (ONRL) 

Defense Documentation Center 
Cameron Station, Building 5 
5010 Duke Street 
Alexandria, Virginia 22314 



89 




(Continued from Inside front cover) 



96 R, C. Atkinson, J. W, Brelsford, And R. M. Shlffrln. Multi-process models for memory with applications to a continuous presentation task. 

April 13, 1966. (J. math, Psychol. , 1967,4, 277-300 ). 

97 P. Suppes and E. Crothers. Some remarks on stimulus-response theories of language learning. June 12, 1966. 

98 R. Bjork. All-or-none subprocesses In the learning of complex sequences, (J. math , Psychol. , 1968, 1 , 182-195). 

99 E. Gammon. The statistical determination of linguistic units. July I, 1966. 

100 P. Suppes, L. Hyman, and M. Jerman. Linear structural models for response and latency performance In arithmetic. In J. P. Hill (ed .) , 

Minnesota Symposia on Child Psychology . Minneapolis, Minn.: 1967. Pp. 160-200). 

101 J. L. Young. Effects of Intervals between reinforcements and test trials In paired-associate learning, August I, 1966. 

102 H. A. Wilson. An Investigation of linguistic unit size In memory processes. August 3, 1966. 

103 J. T. Townsend. Choice behavior In a cued-recognltlon task. August 8, 1966. 

104 W. H. Batchelder. A mathematical analysis of multi-level verbal learning. August 9, 1966. 

105 H. A. Taylor. The observing response In a cued psychophysical task. August 10, 1966. 

106 R. A. Bjork . Learning and short-term retention of paired associates In relation to specific sequences of Interpresentation Intervals. 

August U , 1966. 

107 R. C. Atkinson and R. M. Shlffrln. Some Two-process models for memory. September 30, 1966. 

108 P. Suppes and C. Ihrke. Accelerated program In elementary-school mathematics— the third year. January 30, 1967. 

109 P. Suppes and \. Rosenthal-Hlll. Concept formation by kindergarten children In a card-sorting task. February 27, 1967. 

110 R. C. Atkinson and R. M. Shlffrln. Human memory: a proposed system and Its control processes. March 21 , 1967. 

I J I Theodore S. Rodgers. Linguistic considerations In the design of the Stanford computer-based curriculum In Initial reading. June I, 1967. 

I 12 Jack M. Knutson. Spelling drills using a computer-assisted Instructional system. June 30, 1967. 

I 13 R. C. Atkinson. Instruction in Initial reading under computer control: (he Stanford Project. July 14, 1967. 

I 14 J. W. Brelsford, Jr. and R. C. Atkinson. Recall of paired -associates as a function of overt and covert rehearsal procedures. July 21, 1967. 

I 15 J. H. Stelzer. Some results concerning subjective probability structures with semiorders. August I, 1967 

I 16 0. E. Rumelhart. The effects of Interpresentation Intervals on performance In a continuous paired •associate task. August II, 1967. 

I 17 E. J. Fishman, L. Keller, and R. E. Atkinson. Massed vs. distributed practice In computerized spelling drills. August 18, 1967. 

118 G. J. Groen. An Investigation of some counting algorithms for simple addition problems. August 21 , 1967. 

119 H. A. Wilson and R, C. Atkinson. Computer-based Instruction in Initial reading: a progress report on the Stanford Project. August 25, 1967. 

I2G F. S. Roberts and P. Suppes. Some problems In the geometry of visual perception. August 31, 1967, ( Synthese, 1967, [7» 173-201) 

12 1 0. Jamison. Bayesian decisions under total and partial Ignorance. 0. Jamison and J. Kozleleckl. Subjective probabilities under total 

uncertainty. September 4, 1967. 

122 R. C. Atkinson. Computerized Instruction and the learning process. September 15, 1967. 

123 W. K. Estes. Outline of a theory of punishment. October 1 , 1967. 

124 T. S. Rodgers. Measuring vocabulary difficulty • An analysis of Item variables In learning Russlan-Engllsh and Japanese-Engllsh vocabulary 

parts. December 18, 1967. 

125 W. K. Estes. Reinforcement In human learning. December 20, 1967. * 

126 G. L. Wolford, 0. L.WesseI,W, K. Estes. Further evidence concerning scanning and sampling assumptions of visual detection 

models, January 31 , 1968. 

127 R, C. Atkinson and R. M. Shlffrln. Some speculations on storage and retrieval processes in long-term memory. February 2, 1968. 

128 John Holmgren. Visual detection v/lth Imperfect recognition. March 29, 1968. 

129 Lucille B. Mlodnosky, The Frostig and the Bender Gestalt as predictors of reading achievement. April 12, 1968. 

130 P. Suppes. Some theoretical models for mathematics learning. April 15, 1968. (Journal of Research and Development In Education. 

1967, l, 5-22) * 

131 G. M. Olson, Learning and retention In a continuous recognition task. May 15, 1968. 9 

132 Ruth Norene Hartley. An Investigation of list types and cues to facilitate Initial reading vocabulary acquisition. May 29, 1968. 

133 P. Suppes. Stimulus-response theory of finite automata. June 19, 1968. 

134 N. Moler and P. Suppes. Quantifier-free axioms for constructive plane geometry. June 20, 1968. (In J. C. H. Gerretsen and 

F. Oort (Eds.), Compositlo Mathematlca . VoJ. 20. Groningen, The Netherlands: Wolters-Noordhoff , 1968, Pp. 143-152.) 

135 W, K. Estes and 0, P. Horst. Latency as a function of number or response alternatives In paired-associate learning. July I, 1968. 

136 M. Schlag-Rey and P. Suppes. High-order dimensions In concept identification. July 2, I96B. (Psychom . Scl., 1968, II, 141-142) 

137 R. M. Shlffrln. Search and retrieval processes In long-term memory. August 15, 1968. 

138 R. 0. Freund, G. R. Loftus, and R.C. Atkinson. Applications of muhMprocess models for memory to continuous recognition tasks. 

December 18, 1968. 

139 R. C. Atkinson. Information delay In human learning. December 18, 1968. 

140* R. C. Atkinson, J. E. Holmgren, and J. F. Juola, Processing time as Influenced by the number of elements In the visual display. 

March 14, 1969. 

141 P. Suppes, E. F. Loftus, andM. Jerman, Problem-solving on a computer-based teletype, March 25, 1969, 

142 P. Suppes and Mona Morningstar. Evaluation of three computer-assisted instruction programs. May 2, 1969. 

143 P. Suppes. On the problems of using mathematics In the development of the social sciences. May 12, 1969. 

144 Z. Domotor. Probabilistic relational structures and their applications. May 14, 1969. 

145 R. C. Atkinson and T. 0 . Wlckens. Human memory and the concept of reinforcement. May 20, 1969. 

‘146 R. J. Tltlev. Some model -theoretic results In measurement theory. May 22, 1969. 

147 P. Suppes. Measurement: Problems of theory and application. June 12, 1969. 

148 .'P, Suppes and C. Ihrke * Accelerated Program In elementary-school mathematics— the fourth year, August 7, 1969. 

149 D. Rundus and R.C. Atkinson. Rehearsal In free recall: A procedure for direct observation. August 12, 1969. 

150 P. Suppes and S. Feldman. Young children's comprehension of logical connectives. October 15, 1969. 




( Continued on back cover ) 



( Continued From inside back cover ) 



151 Joaquim H. Laubsch. An adaptive teaching system for optimal item allocation. lW>mBer l4, 1$69. 

152 Roberta L. Klatzky and Richard C. Atkinson. Memory scans based on alternative test stimulus representations. November 25, 1969. 

153 John E. Holmgren. Response latency as an indicant of Information processing in visual search tasks. March 16, 1970. 

154 Patrick Suppes. Probabilistic grammars for natural languages. May 15, 1970. 

155 E. Gammon. A syntactical analysis of some first-grade readers. June 22, 1970. 

156 Kenneth N. Wexler. An automaton analysis of the learning of a nimiature system of Japanese. July 24, 1970. 

157 R. C. Atkinson and J. A. Paulson. An approach to the psychology of instruction. August 14, 1970. 

158 R.C. Atkinson, J.D. Fletcher, H.C. Chetin, and C.M. Stauffer. Instruction in initial reading under computer control: the Stanford project. 

August 13, 1970. / 

159 Dewey J. Rundus. An analysis of rehearsal processes in free recall. August 21, 1970. 

160 R.L. Klatzky, J.F. Juola, and R.C. Atkinson. Test stimulus representation and experimental context effects in memory scanning. 

161 William A. Rottmayer. A formal theory of perception. November 13, 1970. 

162 Elizabeth Jane Fishman Loftus. An analysis of the structural variables that determine problem-solving difficulty on a computer-based teletype. 
December 18, 1970. 

163 Joseph A. Van Campen. Towards the automatic generation of programmed foreign-language instructional materials. January 11, 1971. 

164 Jamesine Friend and R.C. Atkinson. Computer-assisted instruction in programming: AID. January 25, 1971. 

165 Lawrence James Hubert. A formal model for the perceptual processing of geometric configurations. February 19, 1971. 

166 J. F. Juola, I.S. Fischler, C.T. Wood, and R.C. Atkinson. Recognition time for information stored in long-term memory. 

167 R.L. Klatzky and R.C. Atkinson. Specialization of the cerebral hemispheres In scanning for information in short-term memory. 

168 J.D. Fletcher and R.C. Atkinson. An evaluation of the Stanford CAI program in initial reading ( grades K through 3 ). March 12, 1971. 

169 James F. Juola and R.C. Atkinson. Memory scanning for words versus categories. 

170 Ira S. Fischler and James F. Juola. Effects of repeated tests on recognition time for information In long-term memory. 

171 Patrick Suppes. Semantics of context-free fragments of natural languages. March 30, 1971 . 

172 Jamesine Friend. Instruct coders' manual. May 1, 1971. 

173 R.C. Atkinson and R.M. Shiffrin. The control processes of short-term memory. April 19, 1971. 

174 Patrick Suppes. Computer-assisted instruction at Stanford. May 19, 1971. 

175 D. Jamison, J.D. Fletcher, P. Suppes and R. C. Atkinson. Cost and performance of computer-assisted instruction for compensatory education. 

176 Joseph Offlr. Some mathematical models of individual differences in learning and performance. June 28, 1971. 

177 Richard C. Atkinson and James F. Juola. Factors influencing speed and accuracy of word recognition. August 12, 1971. 

17B- P. Suppes, A. Goldberg, G. Kanz, B. Searle and C. Stauffer. Teacher's handbook for CAI courses. September 1, 1971. 

179 Adele Goldberg. A generalized instructional system for elementary mathematical logic. October 11, 1971. 

180 Max Jerman. Instruction in problem solving and an analysis of structural variables that contribute to problem-solving difficulty. November 12, 1971. 

181 Patrick Suppes. On the grammar and model -theoretic semantics of children's noun phrases. November 29, 1971. 

182 Georg Kreisel . Five notes on the application of proof theory to computer science. December 10, 1971 . 

183 James Michael Moloney. An investigation of college student performance on a logic curriculum In a computer-assisted instruction setting. 

January 28, 1972. 

184 J.E. Friend, J.D. Fletcher and R.C. Atkinson. Student performance in computer-assisted instruction in programming. May 10, 1972. 



